Nov 02 14:54:44.420900 ip-10-0-130-229 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 14:54:44.420991 ip-10-0-130-229 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 14:54:44.421002 ip-10-0-130-229 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 14:54:44.421350 ip-10-0-130-229 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 14:54:54.200880 ip-10-0-130-229 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Nov 02 14:54:54.200891 ip-10-0-130-229 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7657d6de108e43fa92abb95ee9a9348b -- Nov 02 14:56:43.546788 ip-10-0-130-229 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:56:43.947583 ip-10-0-130-229 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:56:43.947583 ip-10-0-130-229 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:56:43.947583 ip-10-0-130-229 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:56:43.947583 ip-10-0-130-229 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:56:43.953698 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953614 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.953698 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953695 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.953698 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953699 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.953698 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953701 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953704 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953707 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953710 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953714 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953717 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953720 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953722 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953725 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953727 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953730 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953732 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953735 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953738 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953742 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953744 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953747 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953750 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953752 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953755 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.953817 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953757 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953760 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953763 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953766 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953769 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953771 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953774 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953776 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953779 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953782 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953784 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953787 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953790 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953792 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953796 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953799 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953802 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953804 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953807 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953810 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.954289 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953812 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953815 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953819 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953831 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953835 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953838 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953841 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953844 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953849 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953852 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953854 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953858 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953860 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953864 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953867 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953870 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953873 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.954775 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953875 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953878 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953881 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953883 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953886 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953888 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953891 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953894 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953897 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953899 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953902 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953905 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953908 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953910 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953913 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953915 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953918 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953921 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953923 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953927 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.955192 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953940 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953944 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953946 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953949 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953952 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953954 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953957 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953959 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953962 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953964 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953967 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953969 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953972 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953975 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953977 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953981 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953984 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953986 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953989 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953992 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.955713 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953995 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.953997 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954000 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954003 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954005 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954787 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954794 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954796 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954799 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954802 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954805 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954807 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954811 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954814 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954817 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954819 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954822 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954824 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954827 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.956172 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954830 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954832 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954835 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954837 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954840 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954843 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954845 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954848 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954850 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954853 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954855 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954861 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954866 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954870 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954873 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954876 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954880 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954885 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954888 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.956643 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954891 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954895 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954897 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954900 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954905 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954908 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954911 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954914 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954916 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954919 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954921 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954924 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954926 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954929 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954931 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954934 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954936 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954939 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954942 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.957093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954945 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954947 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954950 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954952 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954955 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954958 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954961 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954964 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954966 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954969 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954971 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954974 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954976 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954979 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954981 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954984 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954987 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954989 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954992 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954994 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.957578 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.954997 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955000 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955003 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955005 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955008 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955011 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955013 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955015 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955018 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955021 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955023 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955026 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955028 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955031 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955033 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955036 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955038 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955041 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955044 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955046 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.958047 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955049 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955052 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955054 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955057 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955059 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955062 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955065 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955067 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955069 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955072 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955074 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955077 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955080 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955910 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955916 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955919 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955921 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955924 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955927 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955929 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.958527 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955932 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955935 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955937 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955940 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955943 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955945 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955948 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955950 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955955 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955958 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955962 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955965 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955968 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955970 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955973 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955975 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955978 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955981 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955983 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955986 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.958997 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955988 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955991 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955994 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955996 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.955998 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956002 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956004 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956007 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956009 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956012 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956014 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956017 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956019 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956022 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956025 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956027 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956030 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956033 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956036 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.959470 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956038 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956041 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956043 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956046 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956049 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956051 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956054 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956057 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956059 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956062 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956064 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956067 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956069 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956071 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956074 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956077 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956079 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956081 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956084 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.959917 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956087 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956089 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956092 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956095 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956097 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956100 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956102 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956105 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956107 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956110 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956112 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956115 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956117 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956121 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956125 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956128 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956131 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956134 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956137 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.960399 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956140 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956143 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956146 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956148 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956151 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956153 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956156 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956158 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956161 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956163 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956165 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956168 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956170 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956175 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956178 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956180 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956183 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956185 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956188 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.960851 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956190 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956193 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956195 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956291 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956307 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956316 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956323 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956328 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956331 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956336 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956340 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956343 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956346 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956349 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956352 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956355 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956358 2580 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956361 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956364 2580 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956367 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956370 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956373 2580 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956376 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956379 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:56:43.961295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956383 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956386 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956390 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956394 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956397 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956401 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956404 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956407 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956410 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956413 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956416 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956420 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956423 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956426 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956432 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956435 2580 flags.go:64] FLAG: --enable-server="true" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956439 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956443 2580 flags.go:64] FLAG: --event-burst="100" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956446 2580 flags.go:64] FLAG: --event-qps="50" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956449 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956452 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956455 2580 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956461 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956464 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956467 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:56:43.961894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956470 2580 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956473 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956476 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956479 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956482 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956485 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956488 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956491 2580 flags.go:64] FLAG: --feature-gates="" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956495 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956498 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956501 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956504 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956508 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956511 2580 flags.go:64] FLAG: --help="false" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956525 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956528 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956533 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956536 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956540 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956543 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956546 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956550 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956553 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:56:43.962612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956556 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956559 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956562 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956565 2580 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956568 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956571 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956574 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956577 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956580 2580 flags.go:64] FLAG: --lock-file="" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956583 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956586 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956589 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956595 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956598 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956601 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956604 2580 flags.go:64] FLAG: --logging-format="text" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956607 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956610 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956613 2580 flags.go:64] FLAG: --manifest-url="" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956616 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956621 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956624 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956634 2580 flags.go:64] FLAG: --max-pods="110" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956638 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956641 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:56:43.963166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956645 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956648 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956651 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956654 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956657 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956669 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956673 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956676 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956679 2580 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956682 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956685 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956688 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956691 2580 flags.go:64] FLAG: --port="10250" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956694 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956697 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a56a95937141b516" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956700 2580 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956703 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956706 2580 flags.go:64] FLAG: --register-node="true" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956709 2580 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956713 2580 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956716 2580 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956719 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956722 2580 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956726 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956729 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956732 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:56:43.963800 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956735 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956738 2580 flags.go:64] FLAG: --runonce="false" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956740 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956743 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956747 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956750 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956754 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956757 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956760 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956764 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956767 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956771 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956774 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956777 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956781 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956784 2580 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956787 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956792 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956795 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956798 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956801 2580 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956804 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956807 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956810 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956813 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:56:43.964649 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956816 2580 flags.go:64] FLAG: --v="2" Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956820 2580 flags.go:64] FLAG: --version="false" Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956824 2580 flags.go:64] FLAG: --vmodule="" Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956828 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.956832 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956954 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956957 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956960 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956963 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956966 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956969 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956971 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956974 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956979 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956981 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956984 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956986 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956989 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956992 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956995 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.956997 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.965304 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957000 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957002 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957005 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957008 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957010 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957013 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957015 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957018 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957021 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957023 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957026 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957028 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957031 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957034 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957036 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957039 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957041 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957044 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957046 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957049 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.965816 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957051 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957054 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957056 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957059 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957063 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957066 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957068 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957071 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957073 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957077 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957080 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957082 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957085 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957088 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957090 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957093 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957096 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957099 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957101 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957104 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.966307 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957107 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957109 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957112 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957114 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957117 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957121 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957125 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957128 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957130 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957133 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957136 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957138 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957141 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957143 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957146 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957148 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957152 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957155 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957158 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.966790 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957160 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957163 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957167 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957169 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957172 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957174 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957177 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957179 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957182 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957185 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957187 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957190 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957192 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957195 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957198 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957200 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957203 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957207 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957210 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957213 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.967243 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957216 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957219 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957221 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957224 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957226 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957229 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957232 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957234 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957237 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957241 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957243 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.957274 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957318 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957323 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957326 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.967730 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957331 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957334 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957336 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957339 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957342 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957344 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957347 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957350 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957352 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957355 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957357 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957360 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957363 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957365 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957368 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957370 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957373 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957375 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957378 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957380 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.968093 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957383 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957385 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957388 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957390 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957393 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957396 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957400 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957402 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957405 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957407 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957410 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957412 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957416 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957419 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957421 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957424 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957427 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957429 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957432 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957434 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.968587 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957437 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957439 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957442 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957444 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957447 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957449 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957452 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957454 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957457 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957459 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957462 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957464 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957467 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957470 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957473 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957475 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957478 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957483 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957489 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.969058 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957492 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957495 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957499 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957503 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957505 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957509 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957512 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957526 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957528 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957531 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957534 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957537 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957539 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957542 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957545 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957547 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957550 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957553 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957555 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957558 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.969575 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957560 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957563 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957565 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957568 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957570 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957573 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957576 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957578 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957581 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957585 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957588 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957593 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957595 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957598 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957601 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957603 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957606 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957611 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957614 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.970055 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957617 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957619 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957622 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957625 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.957627 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.967035 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.967053 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.967867 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.967875 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967932 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967938 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967941 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967944 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967947 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967950 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967952 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967955 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967959 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967961 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.970493 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967964 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967967 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967970 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967973 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967975 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967978 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967981 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967983 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967986 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967989 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967991 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967994 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.967997 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968000 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968003 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968005 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968008 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968010 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968013 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968016 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.970947 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968019 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968023 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968027 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968030 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968033 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968036 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968039 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968041 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968044 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968047 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968049 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968053 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968057 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968060 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968063 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968065 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968068 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968071 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968073 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.971406 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968076 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968079 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968082 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968085 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968087 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968090 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968092 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968095 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968098 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968100 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968103 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968106 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968109 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968112 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968115 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968117 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968120 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968123 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968125 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.971856 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968128 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968131 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968133 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968136 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968139 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968141 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968144 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968147 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968149 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968152 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968155 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968163 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968166 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968169 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968172 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968174 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968177 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968179 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968183 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968186 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.972299 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968188 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968191 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968194 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968196 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968199 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968203 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968205 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968208 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968211 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968213 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968216 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968219 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968221 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968224 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968226 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968229 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968232 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.972776 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968234 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.968264 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968400 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968405 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968408 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968410 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968414 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968418 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968421 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968423 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968426 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968428 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968431 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968434 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968436 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968439 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.973175 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968442 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968444 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968447 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968449 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968452 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968456 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968458 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968461 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968463 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968466 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968469 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968472 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968474 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968477 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968479 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968482 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968484 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968487 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968489 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.973568 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968492 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968495 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968497 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968500 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968502 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968506 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968508 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968511 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968534 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968539 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968542 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968545 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968547 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968550 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968553 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968555 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968558 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968561 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968564 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.974016 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968566 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968569 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968572 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968574 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968577 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968579 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968582 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968585 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968587 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968590 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968593 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968596 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968598 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968601 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968604 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968607 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968609 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968612 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968614 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.974479 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968617 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968620 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968623 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968625 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968628 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968631 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968633 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968636 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968638 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968641 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968644 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968647 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968649 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968652 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968655 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968657 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968660 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968663 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968665 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.974950 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968668 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968670 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968673 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968677 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968680 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968683 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968686 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968688 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968691 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968693 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968697 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968699 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968702 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968704 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968707 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.975391 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968710 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.968734 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968767 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968771 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968775 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968777 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968780 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968782 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968785 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968788 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968790 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968794 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968796 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968799 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968801 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968804 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:56:43.975746 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968806 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968809 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968812 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968814 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968817 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968819 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968822 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968824 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968827 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968831 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968835 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968838 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968841 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968844 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968846 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968849 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968853 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968856 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968858 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:56:43.976134 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968861 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968863 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968866 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968868 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968871 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968873 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968876 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968878 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968881 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968884 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968887 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968889 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968892 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968894 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968897 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968900 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968905 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968908 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968910 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:56:43.976582 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968913 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968915 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968918 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968920 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968923 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968925 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968927 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968930 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968932 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968935 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968942 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968944 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968947 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968950 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968952 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968955 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968957 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968960 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968962 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:56:43.977054 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968965 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968967 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968970 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968974 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968977 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968979 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968982 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968985 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968987 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968990 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968992 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968995 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.968997 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969000 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969002 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969005 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969007 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969010 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969012 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:56:43.977679 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969015 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969017 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969020 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969023 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969025 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969028 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969031 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969035 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969037 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969040 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969042 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969045 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969052 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969055 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:43.969058 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.969118 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.971672 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.972748 2580 server.go:1014] "Starting client certificate rotation" Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.972837 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:56:43.978154 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.972880 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:56:43.978612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.974961 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:56:43.978742 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.978728 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:56:43.993077 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.993059 2580 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:56:43.996966 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.996950 2580 log.go:25] "Validated CRI v1 image API" Nov 02 14:56:43.998186 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:43.998154 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:56:44.001547 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.001527 2580 fs.go:135] Filesystem UUIDs: map[3a3875af-2fe1-4231-a91f-6b9ecf150048:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 83c89c56-23d0-4337-9896-8d171a9f9c2f:/dev/nvme0n1p4] Nov 02 14:56:44.001622 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.001546 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 14:56:44.008666 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.008547 2580 manager.go:217] Machine: {Timestamp:2026-11-02 14:56:44.006052801 +0000 UTC m=+0.369720453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102058 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21bf8278ce86b47cfce9f28bf8964d SystemUUID:ec21bf82-78ce-86b4-7cfc-e9f28bf8964d BootID:7657d6de-108e-43fa-92ab-b95ee9a9348b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e1:8d:7d:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e1:8d:7d:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:89:a3:fd:ce:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:56:44.008666 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.008664 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:56:44.008769 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.008744 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:56:44.010659 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.010640 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:56:44.010784 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.010661 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-229.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:56:44.010823 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.010787 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:56:44.010823 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.010796 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:56:44.010877 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.010835 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:56:44.010877 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.010853 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:56:44.012093 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.012084 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:56:44.016872 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.016860 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:56:44.016979 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.016971 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:56:44.017830 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.017822 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:56:44.017873 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.017834 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:56:44.017873 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.017846 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:56:44.017873 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.017858 2580 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:56:44.017873 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.017866 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:56:44.020300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.020251 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:56:44.024473 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.024454 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:56:44.031581 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.031569 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:56:44.032564 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032552 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:56:44.032602 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032583 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:56:44.032753 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032743 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:56:44.032781 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032760 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:56:44.032781 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032766 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:56:44.032781 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032772 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:56:44.032858 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032782 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:56:44.032858 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.032791 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:56:44.033455 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.033447 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:56:44.033483 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.033455 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:56:44.033483 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.033463 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:56:44.033483 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.033468 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:56:44.033483 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.033476 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:56:44.033614 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.033485 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:56:44.034206 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.034197 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:56:44.034235 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.034209 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:56:44.038350 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.038336 2580 server.go:1263] "Started kubelet" Nov 02 14:56:44.039151 ip-10-0-130-229 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:56:44.040199 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.039426 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:56:44.040199 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.039936 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:56:44.040199 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.040196 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:56:44.040703 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.040687 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:56:44.042964 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.042572 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:56:44.044995 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.044972 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 14:56:44.048606 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.048586 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:56:44.049270 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.049251 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:56:44.049555 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.049500 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:56:44.050028 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050012 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:56:44.050028 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050029 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:56:44.050126 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050013 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:56:44.050126 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050071 2580 factory.go:55] Registering systemd factory Nov 02 14:56:44.050126 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050091 2580 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:56:44.050126 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050110 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:56:44.050126 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050119 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:56:44.050320 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050309 2580 factory.go:153] Registering CRI-O factory Nov 02 14:56:44.050353 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050322 2580 factory.go:223] Registration of the crio container factory successfully Nov 02 14:56:44.050381 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050375 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:56:44.050410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050388 2580 factory.go:103] Registering Raw factory Nov 02 14:56:44.050410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.050399 2580 manager.go:1199] Started watching for new ooms in manager Nov 02 14:56:44.051401 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.051385 2580 manager.go:319] Starting recovery of all containers Nov 02 14:56:44.064292 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.064271 2580 manager.go:324] Recovery completed Nov 02 14:56:44.065582 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.065537 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 02 14:56:44.068893 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.068879 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.070860 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.070833 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.070928 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.070879 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.070928 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.070906 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.071539 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.071508 2580 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:56:44.071539 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.071532 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:56:44.071663 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.071544 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.071663 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.071552 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:56:44.072294 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.072281 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.072351 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.072304 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.072351 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.072314 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.212791 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.149632 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:56:44.212791 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.186359 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-229.us-west-2.compute.internal.18e4424247b01350 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-229.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-229.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:56:44.0382964 +0000 UTC m=+0.401964052,LastTimestamp:2026-11-02 14:56:44.0382964 +0000 UTC m=+0.401964052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-229.us-west-2.compute.internal,}" Nov 02 14:56:44.212791 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.211953 2580 policy_none.go:50] "Start" Nov 02 14:56:44.212791 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.211972 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:56:44.212791 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.211985 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:56:44.214235 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.213968 2580 policy_none.go:44] "Start" Nov 02 14:56:44.235452 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.235224 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-229.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 14:56:44.235584 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.235304 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-229.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 14:56:44.238849 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.238829 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 02 14:56:44.239388 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.239024 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-229.us-west-2.compute.internal.18e4424249a0fb47 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-229.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-229.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-229.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:56:44.070861639 +0000 UTC m=+0.434529305,LastTimestamp:2026-11-02 14:56:44.070861639 +0000 UTC m=+0.434529305,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-229.us-west-2.compute.internal,}" Nov 02 14:56:44.248138 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.248095 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:56:44.250436 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.250412 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:56:44.250526 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.250506 2580 manager.go:341] "Starting Device Plugin manager" Nov 02 14:56:44.250575 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.250541 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:56:44.250575 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.250550 2580 server.go:87] "Starting device plugin registration server" Nov 02 14:56:44.250824 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.250810 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:56:44.250910 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.250826 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:56:44.250964 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.250934 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:56:44.251019 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.251007 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:56:44.251019 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.251018 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:56:44.251628 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.251607 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:56:44.251710 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.251662 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:56:44.257511 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.257477 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:56:44.258676 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.258658 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:56:44.258676 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.258676 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:56:44.258782 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.258694 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:56:44.258782 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.258723 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 14:56:44.266785 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.266763 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:56:44.266849 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.266820 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vnvcp" Nov 02 14:56:44.351098 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.351073 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.351213 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.351075 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:44.352183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.352167 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.352264 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.352195 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.352264 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.352205 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.352264 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.352227 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.360303 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.360285 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal"] Nov 02 14:56:44.360353 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.360343 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.361262 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.361249 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.361317 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.361276 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.361317 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.361285 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.363601 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.363590 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.363748 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.363735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.363783 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.363763 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.364419 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.364405 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.364419 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.364417 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.364506 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.364432 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.364506 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.364438 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.364506 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.364446 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.364609 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.364447 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.366979 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.366964 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.367026 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.366996 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:56:44.370087 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.369938 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:56:44.370087 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.369979 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:56:44.370087 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.369990 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:56:44.405381 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.405362 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.408397 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.408379 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.450330 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.450305 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:44.451626 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.451609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.451682 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.451634 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.451682 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.451654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548a2bc445bef7413f406b710579c42a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"548a2bc445bef7413f406b710579c42a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.451761 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.451685 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:44.452316 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.452300 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.452354 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.452326 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-229.us-west-2.compute.internal\": node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:56:44.549792 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.549728 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.551928 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.551911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.551980 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.551946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.551980 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.551971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548a2bc445bef7413f406b710579c42a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"548a2bc445bef7413f406b710579c42a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.552067 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.552023 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548a2bc445bef7413f406b710579c42a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"548a2bc445bef7413f406b710579c42a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.552067 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.552028 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.552067 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.552028 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.609355 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.609328 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:56:44.609923 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.609911 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.708248 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.708213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.711344 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.711329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:56:44.729963 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.729942 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:56:44.972712 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.972641 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 14:56:44.973250 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:44.972754 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 14:56:44.973250 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:44.972790 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ae753dff0150c41c2b123ad91ad36b56-7d3113d1bc7bb9ad.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-229.us-west-2.compute.internal.18e4424249a1c0dc\": read tcp 10.0.130.229:33674->3.151.193.16:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-229.us-west-2.compute.internal.18e4424249a1c0dc default 4762 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-229.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-229.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-229.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 14:56:44 +0000 UTC,LastTimestamp:2026-11-02 14:56:44.35220863 +0000 UTC m=+0.715876282,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-229.us-west-2.compute.internal,}" Nov 02 14:56:45.019543 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.019499 2580 apiserver.go:52] "Watching apiserver" Nov 02 14:56:45.049763 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.049737 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 14:56:45.270077 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.270001 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:56:45.270318 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.270301 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-pq7wf","openshift-network-operator/iptables-alerter-7f792","kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-kwbcz","openshift-multus/multus-xnz94","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc","openshift-network-diagnostics/network-check-target-hk7f2","openshift-ovn-kubernetes/ovnkube-node-pw4p7","kube-system/global-pull-secret-syncer-88g4t","openshift-dns/node-resolver-znsfq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-wnkp5","openshift-multus/network-metrics-daemon-56kkk","kube-system/konnectivity-agent-48fls"] Nov 02 14:56:45.272019 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.271978 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:51:44 +0000 UTC" deadline="2028-06-05 17:28:15.401711979 +0000 UTC" Nov 02 14:56:45.272067 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.272019 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13946h31m30.129695503s" Nov 02 14:56:45.272829 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.272813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.274939 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.274917 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.276993 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.276977 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.279090 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.279078 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.281299 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.281283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.283763 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.283538 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:45.283763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.283674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:45.285883 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.285871 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.288033 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.288020 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.288085 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.288069 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:45.290212 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.290193 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.293138 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.293117 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.295402 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.295276 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.295402 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.295347 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:45.297975 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.297958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.324241 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.324219 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.324384 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.324265 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.324592 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.324562 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:56:45.324699 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.324612 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlx4j\"" Nov 02 14:56:45.325308 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.325120 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.325308 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.325224 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:56:45.325308 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.325265 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.325308 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.325284 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ct4hh\"" Nov 02 14:56:45.326532 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.326513 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.326655 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.326573 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.326732 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.326601 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnsn9\"" Nov 02 14:56:45.328728 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.328703 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.328805 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.328735 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.328805 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.328784 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:56:45.328873 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.328814 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:56:45.328873 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.328836 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2tgnv\"" Nov 02 14:56:45.330666 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.330649 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.331655 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.331637 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.331758 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.331684 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:56:45.331758 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.331716 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsn9p\"" Nov 02 14:56:45.350817 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.350797 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:56:45.357087 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-system-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.357181 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-kubelet\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.357181 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-host-slash\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.357181 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-conf\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.357309 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357203 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.357309 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357245 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2wv\" (UniqueName: \"kubernetes.io/projected/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-kube-api-access-sl2wv\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.357309 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cnibin\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.357309 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357286 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26d2p\" (UniqueName: \"kubernetes.io/projected/4a16a23a-e4f9-4fd7-a900-28027f31c7de-kube-api-access-26d2p\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.357309 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357308 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991e750-1337-47d8-a151-1c2ed20797f6-hosts-file\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgr68\" (UniqueName: \"kubernetes.io/projected/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-kube-api-access-sgr68\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357340 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a16a23a-e4f9-4fd7-a900-28027f31c7de-serviceca\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357356 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dxc\" (UniqueName: \"kubernetes.io/projected/a08862fb-0cb0-4e65-89b1-739c038bd82f-kube-api-access-x2dxc\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357387 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.357482 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-multus\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-agent-certs\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-konnectivity-ca\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-modprobe-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991e750-1337-47d8-a151-1c2ed20797f6-tmp-dir\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-multus-certs\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357627 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357657 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64862\" (UniqueName: \"kubernetes.io/projected/ce59c590-a124-4655-9aaf-67e37b6e9fc6-kube-api-access-64862\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgbgt\" (UniqueName: \"kubernetes.io/projected/33f48086-60c0-4d87-94a0-4adf4788b971-kube-api-access-lgbgt\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.357759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-log-socket\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8wb\" (UniqueName: \"kubernetes.io/projected/0991e750-1337-47d8-a151-1c2ed20797f6-kube-api-access-5p8wb\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-conf-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-daemon-config\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357897 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-sys-fs\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cni-binary-copy\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-host\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.357997 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-etc-tuned\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-tmp\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-ovn\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358107 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-kubelet-config\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysconfig\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-sys\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358181 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-socket-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-env-overrides\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-netns\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.358295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358317 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-systemd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-node-log\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358363 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-bin\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovn-node-metrics-cert\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358412 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-systemd\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-run\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358446 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-var-lib-kubelet\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358463 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358478 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-config\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358512 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cnibin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a16a23a-e4f9-4fd7-a900-28027f31c7de-host\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358581 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-device-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358598 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-script-lib\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.358985 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-bin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-registration-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-k8s-cni-cncf-io\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-var-lib-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358730 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-etc-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358767 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358794 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-iptables-alerter-script\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-kubelet\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358856 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-systemd-units\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358875 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-netd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqc4\" (UniqueName: \"kubernetes.io/projected/aedd60ea-b38d-4860-ace6-f5ecf7900581-kube-api-access-qcqc4\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-lib-modules\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-socket-dir-parent\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.358978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-hostroot\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359000 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-etc-kubernetes\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359018 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.359487 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359037 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.359937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-dbus\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.359937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffq26\" (UniqueName: \"kubernetes.io/projected/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-kube-api-access-ffq26\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.359937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-kubernetes\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.359937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-slash\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.359937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.359196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-netns\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.423003 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.422985 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 14:56:45.423094 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.423040 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lzsmd" Nov 02 14:56:45.426879 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.426862 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:56:45.426981 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.426970 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:56:45.427095 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427065 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:56:45.427229 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427120 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:56:45.427229 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427118 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.427229 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427146 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.427229 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427145 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:56:45.427229 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427195 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:56:45.427229 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427213 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4crbl\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427310 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427317 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427340 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s2tzl\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427349 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427356 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b9vbw\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427371 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q62fl\"" Nov 02 14:56:45.427497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.427401 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:56:45.460187 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-iptables-alerter-script\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.460275 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460188 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-kubelet\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460275 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-systemd-units\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460275 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-netd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460275 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460249 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-kubelet\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460275 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqc4\" (UniqueName: \"kubernetes.io/projected/aedd60ea-b38d-4860-ace6-f5ecf7900581-kube-api-access-qcqc4\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-lib-modules\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-socket-dir-parent\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-netd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460322 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-systemd-units\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-hostroot\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460387 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-hostroot\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-etc-kubernetes\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460368 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-socket-dir-parent\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460418 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-lib-modules\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460425 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-etc-kubernetes\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460449 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-dbus\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460510 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffq26\" (UniqueName: \"kubernetes.io/projected/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-kube-api-access-ffq26\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-kubernetes\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460554 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-dbus\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460568 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-slash\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460598 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-slash\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-netns\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-kubernetes\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-system-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460644 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-netns\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460661 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-kubelet\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460668 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-system-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-host-slash\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460694 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-iptables-alerter-script\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460703 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-kubelet\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.460888 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460714 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-conf\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-host-slash\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2wv\" (UniqueName: \"kubernetes.io/projected/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-kube-api-access-sl2wv\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cnibin\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26d2p\" (UniqueName: \"kubernetes.io/projected/4a16a23a-e4f9-4fd7-a900-28027f31c7de-kube-api-access-26d2p\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991e750-1337-47d8-a151-1c2ed20797f6-hosts-file\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460842 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgr68\" (UniqueName: \"kubernetes.io/projected/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-kube-api-access-sgr68\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a16a23a-e4f9-4fd7-a900-28027f31c7de-serviceca\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460861 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-conf\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dxc\" (UniqueName: \"kubernetes.io/projected/a08862fb-0cb0-4e65-89b1-739c038bd82f-kube-api-access-x2dxc\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460962 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991e750-1337-47d8-a151-1c2ed20797f6-hosts-file\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460928 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cnibin\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.460992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461010 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.461566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461108 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461129 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-multus\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-agent-certs\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-konnectivity-ca\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-modprobe-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991e750-1337-47d8-a151-1c2ed20797f6-tmp-dir\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461335 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-multus-certs\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461362 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461401 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-multus\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461464 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-modprobe-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461507 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64862\" (UniqueName: \"kubernetes.io/projected/ce59c590-a124-4655-9aaf-67e37b6e9fc6-kube-api-access-64862\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbgt\" (UniqueName: \"kubernetes.io/projected/33f48086-60c0-4d87-94a0-4adf4788b971-kube-api-access-lgbgt\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461617 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-log-socket\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.462396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461671 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8wb\" (UniqueName: \"kubernetes.io/projected/0991e750-1337-47d8-a151-1c2ed20797f6-kube-api-access-5p8wb\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-conf-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-daemon-config\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-sys-fs\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461779 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991e750-1337-47d8-a151-1c2ed20797f6-tmp-dir\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cni-binary-copy\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461818 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-host\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-etc-tuned\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461864 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-tmp\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461886 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-ovn\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461889 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a16a23a-e4f9-4fd7-a900-28027f31c7de-serviceca\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-kubelet-config\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461510 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-multus-certs\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461921 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysconfig\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461963 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-sys\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461979 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.463192 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-socket-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461981 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462008 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-log-socket\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462105 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-socket-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462139 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-env-overrides\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462168 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-netns\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462195 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-host\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462202 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-systemd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462242 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-node-log\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462258 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-sys-fs\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-bin\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovn-node-metrics-cert\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-systemd\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462327 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-conf-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-run\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.463811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-var-lib-kubelet\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462392 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.462420 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-config\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462451 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.462500 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:45.962458889 +0000 UTC m=+2.326126545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cnibin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.462533 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cni-binary-copy\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462587 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-systemd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.462599 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret podName:a3e70c22-8486-4834-8332-a8cf3e34d738 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:45.962587727 +0000 UTC m=+2.326255382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret") pod "global-pull-secret-syncer-88g4t" (UID: "a3e70c22-8486-4834-8332-a8cf3e34d738") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462619 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a16a23a-e4f9-4fd7-a900-28027f31c7de-host\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462636 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-ovn\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462645 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462652 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysconfig\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-device-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-script-lib\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-bin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462739 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cnibin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-kubelet-config\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-registration-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-k8s-cni-cncf-io\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462798 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-sys\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-node-log\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461960 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-konnectivity-ca\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462817 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-var-lib-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462832 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-bin\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462858 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-var-lib-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-etc-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462888 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a16a23a-e4f9-4fd7-a900-28027f31c7de-host\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462895 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-systemd\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462946 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-run\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462986 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-var-lib-kubelet\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.464931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463020 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-env-overrides\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463034 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462700 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-daemon-config\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-device-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463067 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-etc-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.462701 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-netns\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463106 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-registration-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463112 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-bin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.463160 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-k8s-cni-cncf-io\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.461936 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.464054 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-config\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.464177 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-script-lib\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.465249 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-tmp\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.465334 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovn-node-metrics-cert\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.465357 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-etc-tuned\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.465807 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.465507 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-agent-certs\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.518529 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.518498 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:45.518673 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.518537 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:45.518673 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.518551 2580 projected.go:196] Error preparing data for projected volume kube-api-access-grw5d for pod openshift-network-diagnostics/network-check-target-hk7f2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:45.518673 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.518616 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d podName:4d40fd93-7004-40a5-a715-dcbac377dd94 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:46.018603078 +0000 UTC m=+2.382270737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-grw5d" (UniqueName: "kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d") pod "network-check-target-hk7f2" (UID: "4d40fd93-7004-40a5-a715-dcbac377dd94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:45.520255 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.520199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26d2p\" (UniqueName: \"kubernetes.io/projected/4a16a23a-e4f9-4fd7-a900-28027f31c7de-kube-api-access-26d2p\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.521102 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.521069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8wb\" (UniqueName: \"kubernetes.io/projected/0991e750-1337-47d8-a151-1c2ed20797f6-kube-api-access-5p8wb\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.521166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.521114 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffq26\" (UniqueName: \"kubernetes.io/projected/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-kube-api-access-ffq26\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.521166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.521126 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqc4\" (UniqueName: \"kubernetes.io/projected/aedd60ea-b38d-4860-ace6-f5ecf7900581-kube-api-access-qcqc4\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.521433 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.521410 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64862\" (UniqueName: \"kubernetes.io/projected/ce59c590-a124-4655-9aaf-67e37b6e9fc6-kube-api-access-64862\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.523070 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.523048 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2wv\" (UniqueName: \"kubernetes.io/projected/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-kube-api-access-sl2wv\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.525255 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.525236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbgt\" (UniqueName: \"kubernetes.io/projected/33f48086-60c0-4d87-94a0-4adf4788b971-kube-api-access-lgbgt\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.525462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.525446 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dxc\" (UniqueName: \"kubernetes.io/projected/a08862fb-0cb0-4e65-89b1-739c038bd82f-kube-api-access-x2dxc\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.525522 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.525506 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgr68\" (UniqueName: \"kubernetes.io/projected/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-kube-api-access-sgr68\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.596493 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.596463 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:56:45.613273 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.613249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:56:45.634978 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.634956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:56:45.653851 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.653833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xnz94" Nov 02 14:56:45.673766 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.673749 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:56:45.696239 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.696222 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-znsfq" Nov 02 14:56:45.701833 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.701820 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:56:45.717178 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.717159 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:45.719424 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.719406 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:45.720764 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.720746 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:56:45.920409 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:45.920377 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2ff2ec6d97130d91a627ec6e48c555.slice/crio-3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f WatchSource:0}: Error finding container 3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f: Status 404 returned error can't find the container with id 3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f Nov 02 14:56:45.928612 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.928596 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:56:45.933471 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:45.933443 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548a2bc445bef7413f406b710579c42a.slice/crio-0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8 WatchSource:0}: Error finding container 0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8: Status 404 returned error can't find the container with id 0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8 Nov 02 14:56:45.966253 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.966224 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:45.966342 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:45.966258 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:45.966398 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.966384 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:45.966470 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.966455 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:45.966546 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.966462 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:46.966447951 +0000 UTC m=+3.330115591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:45.966601 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:45.966543 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret podName:a3e70c22-8486-4834-8332-a8cf3e34d738 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:46.966509679 +0000 UTC m=+3.330177318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret") pod "global-pull-secret-syncer-88g4t" (UID: "a3e70c22-8486-4834-8332-a8cf3e34d738") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:46.066712 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.066686 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:46.067288 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.066802 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:46.067288 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.066815 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:46.067288 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.066824 2580 projected.go:196] Error preparing data for projected volume kube-api-access-grw5d for pod openshift-network-diagnostics/network-check-target-hk7f2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:46.067288 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.066865 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d podName:4d40fd93-7004-40a5-a715-dcbac377dd94 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:47.066854128 +0000 UTC m=+3.430521768 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-grw5d" (UniqueName: "kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d") pod "network-check-target-hk7f2" (UID: "4d40fd93-7004-40a5-a715-dcbac377dd94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:46.221156 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.221081 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:56:46.262682 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.262642 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"cb2ff2ec6d97130d91a627ec6e48c555","Type":"ContainerStarted","Data":"3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f"} Nov 02 14:56:46.263608 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.263585 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"548a2bc445bef7413f406b710579c42a","Type":"ContainerStarted","Data":"0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8"} Nov 02 14:56:46.426267 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.426210 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:51:45 +0000 UTC" deadline="2028-07-16 01:48:00.61667184 +0000 UTC" Nov 02 14:56:46.426267 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.426260 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14914h51m14.190414152s" Nov 02 14:56:46.565985 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.565955 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08862fb_0cb0_4e65_89b1_739c038bd82f.slice/crio-0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f WatchSource:0}: Error finding container 0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f: Status 404 returned error can't find the container with id 0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f Nov 02 14:56:46.586821 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.586795 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b68b41e_2bcf_4d2c_a881_dcd809b89fc5.slice/crio-cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73 WatchSource:0}: Error finding container cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73: Status 404 returned error can't find the container with id cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73 Nov 02 14:56:46.601445 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.601421 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5067ea99_b14b_4204_a4b4_9cfd0ae2f677.slice/crio-c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc WatchSource:0}: Error finding container c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc: Status 404 returned error can't find the container with id c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc Nov 02 14:56:46.621830 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.621806 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0991e750_1337_47d8_a151_1c2ed20797f6.slice/crio-c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d WatchSource:0}: Error finding container c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d: Status 404 returned error can't find the container with id c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d Nov 02 14:56:46.622446 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.622342 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f48086_60c0_4d87_94a0_4adf4788b971.slice/crio-814925d8dd6b151aa058a78092b9d7bd04cfaee5e902d8b27460bcca786d7f5a WatchSource:0}: Error finding container 814925d8dd6b151aa058a78092b9d7bd04cfaee5e902d8b27460bcca786d7f5a: Status 404 returned error can't find the container with id 814925d8dd6b151aa058a78092b9d7bd04cfaee5e902d8b27460bcca786d7f5a Nov 02 14:56:46.626511 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.626494 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab1bb75_ccb3_4121_b885_4ab456c6ec6d.slice/crio-37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9 WatchSource:0}: Error finding container 37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9: Status 404 returned error can't find the container with id 37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9 Nov 02 14:56:46.627346 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.627330 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a16a23a_e4f9_4fd7_a900_28027f31c7de.slice/crio-7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661 WatchSource:0}: Error finding container 7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661: Status 404 returned error can't find the container with id 7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661 Nov 02 14:56:46.653467 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.653448 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4f0f98_450d_4bdf_aafc_f9eaaebf0f29.slice/crio-aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8 WatchSource:0}: Error finding container aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8: Status 404 returned error can't find the container with id aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8 Nov 02 14:56:46.677818 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:56:46.677801 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedd60ea_b38d_4860_ace6_f5ecf7900581.slice/crio-ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b WatchSource:0}: Error finding container ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b: Status 404 returned error can't find the container with id ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b Nov 02 14:56:46.973396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.973311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:46.973549 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:46.973443 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:46.973549 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.973449 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:46.973549 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.973540 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret podName:a3e70c22-8486-4834-8332-a8cf3e34d738 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:48.973505608 +0000 UTC m=+5.337173261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret") pod "global-pull-secret-syncer-88g4t" (UID: "a3e70c22-8486-4834-8332-a8cf3e34d738") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:46.973699 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.973569 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:46.973699 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:46.973627 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:48.973608864 +0000 UTC m=+5.337276518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:47.074039 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.073989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:47.074632 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.074159 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:47.074632 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.074224 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:47.074632 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.074238 2580 projected.go:196] Error preparing data for projected volume kube-api-access-grw5d for pod openshift-network-diagnostics/network-check-target-hk7f2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:47.074632 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.074299 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d podName:4d40fd93-7004-40a5-a715-dcbac377dd94 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:49.074280913 +0000 UTC m=+5.437948572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-grw5d" (UniqueName: "kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d") pod "network-check-target-hk7f2" (UID: "4d40fd93-7004-40a5-a715-dcbac377dd94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:47.259360 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.259275 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:47.259548 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.259421 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:47.259548 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.259504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:47.259634 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.259592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:47.259687 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.259658 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:47.259750 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:47.259732 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:47.267506 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.267430 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-znsfq" event={"ID":"0991e750-1337-47d8-a151-1c2ed20797f6","Type":"ContainerStarted","Data":"c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d"} Nov 02 14:56:47.268783 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.268757 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xnz94" event={"ID":"5067ea99-b14b-4204-a4b4-9cfd0ae2f677","Type":"ContainerStarted","Data":"c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc"} Nov 02 14:56:47.270211 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.270184 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-48fls" event={"ID":"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5","Type":"ContainerStarted","Data":"cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73"} Nov 02 14:56:47.271788 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.271741 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" event={"ID":"a08862fb-0cb0-4e65-89b1-739c038bd82f","Type":"ContainerStarted","Data":"0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f"} Nov 02 14:56:47.273099 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.273073 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b"} Nov 02 14:56:47.274463 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.274439 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7f792" event={"ID":"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29","Type":"ContainerStarted","Data":"aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8"} Nov 02 14:56:47.275728 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.275706 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pq7wf" event={"ID":"4a16a23a-e4f9-4fd7-a900-28027f31c7de","Type":"ContainerStarted","Data":"7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661"} Nov 02 14:56:47.277004 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.276982 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerStarted","Data":"37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9"} Nov 02 14:56:47.279407 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.279328 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" event={"ID":"33f48086-60c0-4d87-94a0-4adf4788b971","Type":"ContainerStarted","Data":"814925d8dd6b151aa058a78092b9d7bd04cfaee5e902d8b27460bcca786d7f5a"} Nov 02 14:56:47.426646 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.426601 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:51:45 +0000 UTC" deadline="2028-08-16 03:32:17.694242078 +0000 UTC" Nov 02 14:56:47.426646 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:47.426643 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15660h35m30.267603793s" Nov 02 14:56:48.991919 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:48.991881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:48.992397 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:48.991935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:48.992397 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:48.992068 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:48.992397 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:48.992126 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret podName:a3e70c22-8486-4834-8332-a8cf3e34d738 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:52.992109793 +0000 UTC m=+9.355777445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret") pod "global-pull-secret-syncer-88g4t" (UID: "a3e70c22-8486-4834-8332-a8cf3e34d738") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:48.992397 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:48.992198 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:48.992397 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:48.992234 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:52.992221873 +0000 UTC m=+9.355889528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:49.093012 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:49.092974 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:49.093208 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.093193 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:49.093277 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.093217 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:49.093277 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.093230 2580 projected.go:196] Error preparing data for projected volume kube-api-access-grw5d for pod openshift-network-diagnostics/network-check-target-hk7f2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:49.093382 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.093287 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d podName:4d40fd93-7004-40a5-a715-dcbac377dd94 nodeName:}" failed. No retries permitted until 2026-11-02 14:56:53.093270855 +0000 UTC m=+9.456938510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-grw5d" (UniqueName: "kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d") pod "network-check-target-hk7f2" (UID: "4d40fd93-7004-40a5-a715-dcbac377dd94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:49.259759 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:49.259540 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:49.259901 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.259876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:49.259975 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:49.259958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:49.260046 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.260026 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:49.260115 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:49.260088 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:49.260177 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:49.260157 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:50.302626 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:50.302548 2580 generic.go:356] "Generic (PLEG): container finished" podID="cb2ff2ec6d97130d91a627ec6e48c555" containerID="0b12ae965bd7ed6219b15e43661aa1a8a014b87a1f2f07b8c10da41957457141" exitCode=0 Nov 02 14:56:50.303072 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:50.302639 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"cb2ff2ec6d97130d91a627ec6e48c555","Type":"ContainerDied","Data":"0b12ae965bd7ed6219b15e43661aa1a8a014b87a1f2f07b8c10da41957457141"} Nov 02 14:56:50.309858 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:50.309772 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="3c7f3fbce41fd510fcd3ff08fea84281139490074ef2dc2786a23333158d7b34" exitCode=0 Nov 02 14:56:50.309858 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:50.309842 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerDied","Data":"3c7f3fbce41fd510fcd3ff08fea84281139490074ef2dc2786a23333158d7b34"} Nov 02 14:56:50.328949 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:50.328432 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"548a2bc445bef7413f406b710579c42a","Type":"ContainerStarted","Data":"49a7681841195eaf95a5d8050844e0f4a98c7724ccd4db55e02a5627079eeab5"} Nov 02 14:56:51.259486 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:51.259451 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:51.259653 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:51.259623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:51.259731 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:51.259719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:51.259799 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:51.259785 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:51.259856 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:51.259834 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:51.259911 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:51.259893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:51.337669 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:51.337634 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"cb2ff2ec6d97130d91a627ec6e48c555","Type":"ContainerStarted","Data":"2dcce3ae79a83975bbc675f3150d09471cf45a9a87f54daee81dcc3d456cff51"} Nov 02 14:56:51.447204 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:51.447114 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" podStartSLOduration=7.447088422 podStartE2EDuration="7.447088422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:50.640592538 +0000 UTC m=+7.004260214" watchObservedRunningTime="2026-11-02 14:56:51.447088422 +0000 UTC m=+7.810756083" Nov 02 14:56:52.344061 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:52.344029 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-48fls" event={"ID":"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5","Type":"ContainerStarted","Data":"56d2a655436c720561f7887f866569c359daa66ab8b783a0af0edacf432673ed"} Nov 02 14:56:52.452528 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:52.452466 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" podStartSLOduration=8.452450666 podStartE2EDuration="8.452450666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:51.447808071 +0000 UTC m=+7.811475750" watchObservedRunningTime="2026-11-02 14:56:52.452450666 +0000 UTC m=+8.816118327" Nov 02 14:56:53.031699 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.031668 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:53.031837 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.031721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:53.031897 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.031848 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:53.031946 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.031910 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret podName:a3e70c22-8486-4834-8332-a8cf3e34d738 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:01.031891617 +0000 UTC m=+17.395559274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret") pod "global-pull-secret-syncer-88g4t" (UID: "a3e70c22-8486-4834-8332-a8cf3e34d738") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:56:53.032009 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.031977 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:53.032063 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.032011 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:01.032000145 +0000 UTC m=+17.395667797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:56:53.132207 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.132164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:53.132365 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.132301 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:56:53.132365 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.132318 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:56:53.132365 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.132328 2580 projected.go:196] Error preparing data for projected volume kube-api-access-grw5d for pod openshift-network-diagnostics/network-check-target-hk7f2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:53.132470 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.132389 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d podName:4d40fd93-7004-40a5-a715-dcbac377dd94 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:01.13236756 +0000 UTC m=+17.496035199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-grw5d" (UniqueName: "kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d") pod "network-check-target-hk7f2" (UID: "4d40fd93-7004-40a5-a715-dcbac377dd94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:56:53.259303 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.259224 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:53.259446 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.259359 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:53.259446 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.259426 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:53.259536 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.259500 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:53.259591 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.259563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:53.259663 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:53.259645 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:53.348372 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.348331 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pq7wf" event={"ID":"4a16a23a-e4f9-4fd7-a900-28027f31c7de","Type":"ContainerStarted","Data":"5fe022add2267b9738d2b3a5c2d2021ddbc3684a1b86fc58b078b1f40cf6e153"} Nov 02 14:56:53.351765 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.351738 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" event={"ID":"a08862fb-0cb0-4e65-89b1-739c038bd82f","Type":"ContainerStarted","Data":"67ec82f8109e6b7df15da271ce9bb8302161cbab67aaf2aeb1f60167cc352a21"} Nov 02 14:56:53.461451 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:53.461392 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-48fls" podStartSLOduration=4.841254323 podStartE2EDuration="9.461372566s" totalImagesPullingTime="4.620118243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:52.453610583 +0000 UTC m=+8.817278245" watchObservedRunningTime="2026-11-02 14:56:53.461372566 +0000 UTC m=+9.825040221" Nov 02 14:56:55.260723 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.259991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:55.260723 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:55.260349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:55.260723 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.260472 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:55.263213 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:55.260633 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:55.263213 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.260857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:55.263213 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:55.262490 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:55.284869 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.284736 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:56:55.359688 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.359641 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-znsfq" event={"ID":"0991e750-1337-47d8-a151-1c2ed20797f6","Type":"ContainerStarted","Data":"b85443bc89a3fe216e738fb502487e749928a6806a6d2dcb38696ebfe6e3a17a"} Nov 02 14:56:55.362882 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.362372 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xnz94" event={"ID":"5067ea99-b14b-4204-a4b4-9cfd0ae2f677","Type":"ContainerStarted","Data":"488c39af0585753bb395d99407448a0070106312872db099467a1a84b849aec6"} Nov 02 14:56:55.365620 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.365598 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" event={"ID":"a08862fb-0cb0-4e65-89b1-739c038bd82f","Type":"ContainerStarted","Data":"f4cf0877ea617204576b4ea3c4c2f080703f5a3d1046a857b42845d530ca82fa"} Nov 02 14:56:55.368554 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.367996 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerStarted","Data":"9c726baec89f8bd72fd45bec31259775891fb236870a218eff778a280b916ec8"} Nov 02 14:56:55.469725 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.469664 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-znsfq" podStartSLOduration=3.004035863 podStartE2EDuration="11.469645319s" totalImagesPullingTime="8.465609456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:55.469354054 +0000 UTC m=+11.833021718" watchObservedRunningTime="2026-11-02 14:56:55.469645319 +0000 UTC m=+11.833312985" Nov 02 14:56:55.469989 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.469956 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pq7wf" podStartSLOduration=5.629049363 podStartE2EDuration="11.469949086s" totalImagesPullingTime="5.840899723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:53.463404703 +0000 UTC m=+9.827072363" watchObservedRunningTime="2026-11-02 14:56:55.469949086 +0000 UTC m=+11.833616749" Nov 02 14:56:55.582609 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:55.582556 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-xnz94" podStartSLOduration=3.6603830630000003 podStartE2EDuration="11.582541109s" totalImagesPullingTime="7.922158046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:55.582245561 +0000 UTC m=+11.945913222" watchObservedRunningTime="2026-11-02 14:56:55.582541109 +0000 UTC m=+11.946208767" Nov 02 14:56:56.261931 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:56.261809 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:56:55.284751854Z","UUID":"4680f7a6-c456-400b-b2b5-f694d6d9e0b4","Handler":null,"Name":"","Endpoint":""} Nov 02 14:56:56.263497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:56.263471 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:56:56.263497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:56.263502 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:56:56.373561 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:56.373476 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="9c726baec89f8bd72fd45bec31259775891fb236870a218eff778a280b916ec8" exitCode=0 Nov 02 14:56:56.374437 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:56.374406 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerDied","Data":"9c726baec89f8bd72fd45bec31259775891fb236870a218eff778a280b916ec8"} Nov 02 14:56:57.161879 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.161795 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:57.162305 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.162273 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:56:57.259407 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.259373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:57.259622 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:57.259593 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:57.259946 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.259930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:57.260029 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:57.260012 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:56:57.260069 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.260055 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:57.260120 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:57.260105 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:57.380544 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.380495 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" event={"ID":"a08862fb-0cb0-4e65-89b1-739c038bd82f","Type":"ContainerStarted","Data":"80b0544d0b3e770dd1b71d3d196345dddebe509cbe7c71c9d1fba63584c467a4"} Nov 02 14:56:57.382751 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.382723 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7f792" event={"ID":"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29","Type":"ContainerStarted","Data":"9a75cca7db31d13ff4c77264747d0c9d5b415cdbf6a9988d689f559589f2c2ea"} Nov 02 14:56:57.499554 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:57.499440 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" podStartSLOduration=3.618598657 podStartE2EDuration="13.499424627s" totalImagesPullingTime="9.88082597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:57.493657145 +0000 UTC m=+13.857324806" watchObservedRunningTime="2026-11-02 14:56:57.499424627 +0000 UTC m=+13.863092290" Nov 02 14:56:58.387780 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:58.387743 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="b6b19e3db97e5521de6c6f037405678a4f28d932bd6d4634459b7f31b50b95a1" exitCode=0 Nov 02 14:56:58.388396 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:58.387831 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerDied","Data":"b6b19e3db97e5521de6c6f037405678a4f28d932bd6d4634459b7f31b50b95a1"} Nov 02 14:56:58.389501 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:58.389448 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" event={"ID":"33f48086-60c0-4d87-94a0-4adf4788b971","Type":"ContainerStarted","Data":"83644f911d581cd66dc4b74a2f2562e63bc14e77f76ab74099fe5a07e78d43ee"} Nov 02 14:56:58.603043 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:58.602975 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7f792" podStartSLOduration=5.946947495 podStartE2EDuration="14.602959689s" totalImagesPullingTime="8.656012194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:57.623821247 +0000 UTC m=+13.987488912" watchObservedRunningTime="2026-11-02 14:56:58.602959689 +0000 UTC m=+14.966627349" Nov 02 14:56:58.713012 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:58.712926 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" podStartSLOduration=3.749885598 podStartE2EDuration="14.712912553s" totalImagesPullingTime="10.963026955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:56:58.712387197 +0000 UTC m=+15.076054857" watchObservedRunningTime="2026-11-02 14:56:58.712912553 +0000 UTC m=+15.076580247" Nov 02 14:56:59.259829 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:59.259793 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:56:59.259996 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:59.259915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:56:59.259996 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:59.259926 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:56:59.259996 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:56:59.259975 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:56:59.260107 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:59.260051 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:56:59.260137 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:56:59.260108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:00.396072 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:00.395868 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="1d37ee1085995bcd8e59237cb6a3a0fd537e0c2437900e65d0a749889281880a" exitCode=0 Nov 02 14:57:00.396072 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:00.395952 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerDied","Data":"1d37ee1085995bcd8e59237cb6a3a0fd537e0c2437900e65d0a749889281880a"} Nov 02 14:57:01.092589 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.092558 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:01.092589 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.092592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:01.092771 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.092690 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:01.092771 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.092696 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:01.092771 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.092748 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret podName:a3e70c22-8486-4834-8332-a8cf3e34d738 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:17.092733361 +0000 UTC m=+33.456401017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret") pod "global-pull-secret-syncer-88g4t" (UID: "a3e70c22-8486-4834-8332-a8cf3e34d738") : object "kube-system"/"original-pull-secret" not registered Nov 02 14:57:01.092771 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.092767 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:17.092757638 +0000 UTC m=+33.456425278 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 14:57:01.193457 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.193434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:01.193646 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.193630 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 14:57:01.193701 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.193650 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 14:57:01.193701 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.193659 2580 projected.go:196] Error preparing data for projected volume kube-api-access-grw5d for pod openshift-network-diagnostics/network-check-target-hk7f2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:01.193771 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.193702 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d podName:4d40fd93-7004-40a5-a715-dcbac377dd94 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:17.193689621 +0000 UTC m=+33.557357272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-grw5d" (UniqueName: "kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d") pod "network-check-target-hk7f2" (UID: "4d40fd93-7004-40a5-a715-dcbac377dd94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 14:57:01.259317 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.259293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:01.259412 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.259293 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:01.259465 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.259424 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:01.259539 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.259295 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:01.259539 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.259526 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:01.259647 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:01.259617 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:01.399772 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.399618 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"989960ab507fd9babd2fdeca0974dc23f235d7263de7df288ac391010962377a"} Nov 02 14:57:01.400043 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.399822 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"15939c2ad854736c12a11b8c1d363d1b397b1a90e1e5f1f229450014a4ad02ae"} Nov 02 14:57:01.400043 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.399854 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"30ef546686e602114742549f6f201ff5b6a2463beb6429e612fbe8a3c4ba0699"} Nov 02 14:57:01.400043 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:01.399881 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"e2d069e17aeb074beee41223465197fbf7b18512d31325bacfb0ed09a00f38f9"} Nov 02 14:57:02.404920 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:02.404886 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"87e44576ef0068a97108410b5eb9328783ba95df1b7902908188ac65487b2e4e"} Nov 02 14:57:02.405370 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:02.404951 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"5aca5d44067e5f7fd3bef1ee0f780bc6adbb8e0d342108f14961989e73116020"} Nov 02 14:57:03.259592 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:03.259561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:03.259730 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:03.259561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:03.259730 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:03.259688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:03.259809 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:03.259654 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:03.259809 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:03.259756 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:03.259874 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:03.259847 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:03.409947 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:03.409922 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="5c4471d57b76be2c0b6f31a2cf1f8e99ecfda267b94e5dcfbdadd30071e1bbae" exitCode=0 Nov 02 14:57:03.410457 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:03.409955 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerDied","Data":"5c4471d57b76be2c0b6f31a2cf1f8e99ecfda267b94e5dcfbdadd30071e1bbae"} Nov 02 14:57:04.414887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:04.414851 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"e9849a8e125350589f7578311a82b74c314a993dbbc42391d71062f844d4b45b"} Nov 02 14:57:04.417568 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:04.417541 2580 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="3f6543a83842760e5ca2aab9f2bb5aba2e3620cf00dcdc5c7b7884282bf02ea4" exitCode=0 Nov 02 14:57:04.417668 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:04.417579 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerDied","Data":"3f6543a83842760e5ca2aab9f2bb5aba2e3620cf00dcdc5c7b7884282bf02ea4"} Nov 02 14:57:05.259907 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:05.259874 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:05.260073 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:05.259921 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:05.260073 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:05.259982 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:05.260073 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:05.260039 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:05.260168 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:05.260089 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:05.260168 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:05.260146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:05.423148 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:05.423117 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" event={"ID":"fab1bb75-ccb3-4121-b885-4ab456c6ec6d","Type":"ContainerStarted","Data":"4b03e1bee798b4ef35f639834519fd532a2abc54f98ae45e443f0e3c3069a794"} Nov 02 14:57:05.539979 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:05.539890 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wnkp5" podStartSLOduration=8.826523629 podStartE2EDuration="21.539877612s" totalImagesPullingTime="12.713353983s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:05.539557827 +0000 UTC m=+21.903225490" watchObservedRunningTime="2026-11-02 14:57:05.539877612 +0000 UTC m=+21.903545280" Nov 02 14:57:06.428271 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:06.428073 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" event={"ID":"aedd60ea-b38d-4860-ace6-f5ecf7900581","Type":"ContainerStarted","Data":"64f00399f955a5347bff459cace59774bae19ba5d82184e8bd19cd6f4772de91"} Nov 02 14:57:06.687070 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:06.686978 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" podStartSLOduration=8.198318809 podStartE2EDuration="22.686965442s" totalImagesPullingTime="14.488646633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:06.686552712 +0000 UTC m=+23.050220375" watchObservedRunningTime="2026-11-02 14:57:06.686965442 +0000 UTC m=+23.050633120" Nov 02 14:57:07.259831 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.259798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:07.259991 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.259839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:07.259991 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.259798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:07.259991 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:07.259912 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:07.259991 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:07.259980 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:07.260130 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:07.260050 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:07.429779 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.429755 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:07.430172 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.430154 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:07.430241 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.430220 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:07.445965 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.445944 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:07.446166 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:07.446153 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:08.325183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.325153 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:08.325547 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.325290 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:08.325732 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.325709 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:08.411932 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.411899 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hk7f2"] Nov 02 14:57:08.412084 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.412016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:08.412123 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:08.412107 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:08.414810 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.414782 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88g4t"] Nov 02 14:57:08.414946 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.414899 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:08.415024 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:08.415001 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:08.415495 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.415469 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-56kkk"] Nov 02 14:57:08.415602 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.415577 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:08.415673 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:08.415657 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:08.431496 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:08.431470 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:09.433203 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:09.433175 2580 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:09.799627 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:09.799485 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:10.259316 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:10.259289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:10.259497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:10.259289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:10.259497 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:10.259400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:10.259497 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:10.259432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:10.259696 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:10.259541 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:10.259696 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:10.259592 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:10.448336 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:10.448288 2580 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" podUID="aedd60ea-b38d-4860-ace6-f5ecf7900581" containerName="ovnkube-controller" probeResult="failure" output="" Nov 02 14:57:12.259205 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:12.259172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:12.259205 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:12.259205 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:12.259731 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:12.259275 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:12.259731 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:12.259346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:12.259731 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:12.259436 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:12.259731 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:12.259448 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:14.260279 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.260091 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:14.260643 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.260335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hk7f2" podUID="4d40fd93-7004-40a5-a715-dcbac377dd94" Nov 02 14:57:14.260643 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.260170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:14.260643 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.260196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:14.260643 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.260430 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-56kkk" podUID="ce59c590-a124-4655-9aaf-67e37b6e9fc6" Nov 02 14:57:14.260643 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.260504 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88g4t" podUID="a3e70c22-8486-4834-8332-a8cf3e34d738" Nov 02 14:57:14.478798 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.478724 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeReady" Nov 02 14:57:14.478957 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.478847 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 14:57:14.622432 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.622381 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7hpmc"] Nov 02 14:57:14.625063 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.625047 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wfz5w"] Nov 02 14:57:14.625183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.625165 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:14.627033 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.627015 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f"] Nov 02 14:57:14.627161 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.627144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:14.629035 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.629016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.632750 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.632697 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59c9b7fcd6-bxrc7"] Nov 02 14:57:14.635047 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.635028 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7"] Nov 02 14:57:14.635189 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.635168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.637416 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.637400 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54fb4b65f9-w4zp9"] Nov 02 14:57:14.637527 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.637502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.639669 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.639654 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f"] Nov 02 14:57:14.639771 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.639759 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.641948 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.641934 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf"] Nov 02 14:57:14.642048 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.642036 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:14.644562 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.644460 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.668875 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.668855 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84798d88f5-xqqrs"] Nov 02 14:57:14.674145 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.673770 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8vstz"] Nov 02 14:57:14.674145 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.673853 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.677207 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.677189 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-r6gt7"] Nov 02 14:57:14.677302 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.677288 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678310 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678347 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678386 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678393 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678457 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678458 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-chwdw\"" Nov 02 14:57:14.679088 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.678503 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.680697 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.680678 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nsrp7"] Nov 02 14:57:14.680950 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.680930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.683646 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.683622 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 14:57:14.683745 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.683655 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z"] Nov 02 14:57:14.683745 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.683719 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:14.683852 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.683813 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58zgw\"" Nov 02 14:57:14.683982 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.683948 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 14:57:14.685723 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.685703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc4a9361-8679-4e65-83f4-d8d876321372-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.685811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.685728 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hvqx\" (UniqueName: \"kubernetes.io/projected/cc4a9361-8679-4e65-83f4-d8d876321372-kube-api-access-2hvqx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.685811 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.685765 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhxb4\" (UniqueName: \"kubernetes.io/projected/a88a6270-a93e-487d-a0b2-53a5b020c388-kube-api-access-qhxb4\") pod \"kas-connection-checker-685895c7c9-7hpmc\" (UID: \"a88a6270-a93e-487d-a0b2-53a5b020c388\") " pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:14.685978 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.685863 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77zm\" (UniqueName: \"kubernetes.io/projected/e66a1a72-e152-44d7-aa2b-846e8a872078-kube-api-access-q77zm\") pod \"kas-connection-checker-685895c7c9-wfz5w\" (UID: \"e66a1a72-e152-44d7-aa2b-846e8a872078\") " pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:14.685978 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.685910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc4a9361-8679-4e65-83f4-d8d876321372-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.686476 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.686459 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql"] Nov 02 14:57:14.686622 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.686603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:14.689086 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.689070 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7s4b6"] Nov 02 14:57:14.689209 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.689179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.691876 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.691859 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5xk7d"] Nov 02 14:57:14.692158 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.692136 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:14.694696 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.694677 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-r4vdl"] Nov 02 14:57:14.695234 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.695209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.699374 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.699356 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7hpmc"] Nov 02 14:57:14.699471 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.699394 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wfz5w"] Nov 02 14:57:14.699471 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.699416 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f"] Nov 02 14:57:14.699648 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.699482 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.714465 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.714448 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 14:57:14.730186 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.730161 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 14:57:14.731420 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.731392 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7"] Nov 02 14:57:14.732456 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.732438 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf"] Nov 02 14:57:14.733480 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.733465 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59c9b7fcd6-bxrc7"] Nov 02 14:57:14.734313 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.734296 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nsrp7"] Nov 02 14:57:14.735173 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.735155 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8vstz"] Nov 02 14:57:14.742344 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.742320 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4vdl"] Nov 02 14:57:14.745849 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.745827 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5xk7d"] Nov 02 14:57:14.761382 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.761362 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f"] Nov 02 14:57:14.761983 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.761967 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84798d88f5-xqqrs"] Nov 02 14:57:14.762552 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.762539 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54fb4b65f9-w4zp9"] Nov 02 14:57:14.763087 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.763070 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql"] Nov 02 14:57:14.763656 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.763640 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7s4b6"] Nov 02 14:57:14.774918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.774898 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775032 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775020 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775146 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775128 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:57:14.775218 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775163 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775218 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775179 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 14:57:14.775322 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775250 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 14:57:14.775322 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775256 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 14:57:14.775322 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775258 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775317 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775294 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775264 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775345 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 14:57:14.775462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775411 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775706 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775501 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775706 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775510 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775706 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775608 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775706 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775616 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775706 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775689 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775715 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775719 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775742 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775772 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775786 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775867 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:14.775918 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.775873 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:14.776642 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.776629 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 14:57:14.778005 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.777992 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 14:57:14.786424 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e6e6852-e747-4869-aea2-5e0bbc44988f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.786508 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.786508 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786450 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-tmp\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.786508 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.786632 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6d3f56-0c8f-4631-aed3-7dedc802377a-tmp-dir\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.786632 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786603 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hn6p\" (UniqueName: \"kubernetes.io/projected/8f6d3f56-0c8f-4631-aed3-7dedc802377a-kube-api-access-4hn6p\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.786705 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.786738 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.786788 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786761 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffg8g\" (UniqueName: \"kubernetes.io/projected/651ad3fc-4794-4a59-b41f-4fb01ed533d5-kube-api-access-ffg8g\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.786839 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.786883 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786842 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd9fb448-f983-484e-85a4-afbd3efddd1d-serving-cert\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.786883 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.786859 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.787032 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.787093 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbc4v\" (UniqueName: \"kubernetes.io/projected/13c7b469-cf39-4d61-8f78-6fe03ba07dde-kube-api-access-rbc4v\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.787093 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.787093 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787073 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.787214 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc4a9361-8679-4e65-83f4-d8d876321372-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.787214 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787137 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-stats-auth\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.787214 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvt6\" (UniqueName: \"kubernetes.io/projected/108f7147-992f-44ce-8cc4-dd7f8cd2521d-kube-api-access-mbvt6\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.787214 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787178 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13c7b469-cf39-4d61-8f78-6fe03ba07dde-serving-cert\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787222 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-snapshots\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsj84\" (UniqueName: \"kubernetes.io/projected/66c2abab-7f39-4d34-8571-ad883dde9532-kube-api-access-bsj84\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87mw\" (UniqueName: \"kubernetes.io/projected/dd9fb448-f983-484e-85a4-afbd3efddd1d-kube-api-access-c87mw\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.787404 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787398 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.787777 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.787777 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.787777 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787631 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc4a9361-8679-4e65-83f4-d8d876321372-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.788175 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.787609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-service-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.788241 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66c2abab-7f39-4d34-8571-ad883dde9532-serving-cert\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.788241 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45cgf\" (UniqueName: \"kubernetes.io/projected/f264a363-68bf-4fc1-87c6-824a8796d7ef-kube-api-access-45cgf\") pod \"kas-connection-checker-685895c7c9-7s4b6\" (UID: \"f264a363-68bf-4fc1-87c6-824a8796d7ef\") " pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:14.788323 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.788323 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.788323 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788308 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77zm\" (UniqueName: \"kubernetes.io/projected/e66a1a72-e152-44d7-aa2b-846e8a872078-kube-api-access-q77zm\") pod \"kas-connection-checker-685895c7c9-wfz5w\" (UID: \"e66a1a72-e152-44d7-aa2b-846e8a872078\") " pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:14.788448 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788336 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-config\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.788448 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.788573 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788558 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvwn\" (UniqueName: \"kubernetes.io/projected/f1684862-0c18-4f2f-87fa-811ca427d4a5-kube-api-access-mjvwn\") pod \"volume-data-source-validator-58c9f5f46d-k262f\" (UID: \"f1684862-0c18-4f2f-87fa-811ca427d4a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:14.788631 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scq68\" (UniqueName: \"kubernetes.io/projected/f5cdff40-3fc5-423b-b14f-21172fab4f71-kube-api-access-scq68\") pod \"network-check-source-6f78d59f99-p8v6z\" (UID: \"f5cdff40-3fc5-423b-b14f-21172fab4f71\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:14.788681 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.788681 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788665 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.788771 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.788771 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788739 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f742m\" (UniqueName: \"kubernetes.io/projected/8e6e6852-e747-4869-aea2-5e0bbc44988f-kube-api-access-f742m\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.788771 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788760 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:14.788916 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/651ad3fc-4794-4a59-b41f-4fb01ed533d5-cert\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.788916 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.788916 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788871 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13c7b469-cf39-4d61-8f78-6fe03ba07dde-config\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.788916 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788899 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/77135550-06c2-4f6b-8b7a-9ae0f81abb70-kube-api-access-nzpbn\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.788933 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6d3f56-0c8f-4631-aed3-7dedc802377a-config-volume\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789018 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc4a9361-8679-4e65-83f4-d8d876321372-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvqx\" (UniqueName: \"kubernetes.io/projected/cc4a9361-8679-4e65-83f4-d8d876321372-kube-api-access-2hvqx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxb4\" (UniqueName: \"kubernetes.io/projected/a88a6270-a93e-487d-a0b2-53a5b020c388-kube-api-access-qhxb4\") pod \"kas-connection-checker-685895c7c9-7hpmc\" (UID: \"a88a6270-a93e-487d-a0b2-53a5b020c388\") " pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-default-certificate\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.789150 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789147 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.789358 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.789170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.793190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.793166 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc4a9361-8679-4e65-83f4-d8d876321372-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.820455 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820426 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 14:57:14.820765 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820745 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 14:57:14.820838 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820750 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 14:57:14.820838 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820771 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 14:57:14.820838 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820773 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 14:57:14.820838 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820785 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 14:57:14.820838 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820815 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 14:57:14.820838 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820824 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 14:57:14.821041 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.820878 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4xxxf\"" Nov 02 14:57:14.821275 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821259 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 14:57:14.821334 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821277 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:57:14.821334 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821287 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:57:14.821334 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821299 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pr6dp\"" Nov 02 14:57:14.821432 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821341 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 14:57:14.821432 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821408 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w8kzp\"" Nov 02 14:57:14.821432 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821418 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r6rrw\"" Nov 02 14:57:14.821563 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821442 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d5546\"" Nov 02 14:57:14.821563 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821462 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cr8rg\"" Nov 02 14:57:14.821563 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.821487 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4d7td\"" Nov 02 14:57:14.822327 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.822311 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z"] Nov 02 14:57:14.822896 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.822882 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-r6gt7"] Nov 02 14:57:14.846327 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.846305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxb4\" (UniqueName: \"kubernetes.io/projected/a88a6270-a93e-487d-a0b2-53a5b020c388-kube-api-access-qhxb4\") pod \"kas-connection-checker-685895c7c9-7hpmc\" (UID: \"a88a6270-a93e-487d-a0b2-53a5b020c388\") " pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:14.846412 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.846343 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvqx\" (UniqueName: \"kubernetes.io/projected/cc4a9361-8679-4e65-83f4-d8d876321372-kube-api-access-2hvqx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.846412 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.846310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77zm\" (UniqueName: \"kubernetes.io/projected/e66a1a72-e152-44d7-aa2b-846e8a872078-kube-api-access-q77zm\") pod \"kas-connection-checker-685895c7c9-wfz5w\" (UID: \"e66a1a72-e152-44d7-aa2b-846e8a872078\") " pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:14.889815 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.889943 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889820 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbc4v\" (UniqueName: \"kubernetes.io/projected/13c7b469-cf39-4d61-8f78-6fe03ba07dde-kube-api-access-rbc4v\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.889943 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.889943 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.889943 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-stats-auth\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.889943 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889925 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvt6\" (UniqueName: \"kubernetes.io/projected/108f7147-992f-44ce-8cc4-dd7f8cd2521d-kube-api-access-mbvt6\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889952 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13c7b469-cf39-4d61-8f78-6fe03ba07dde-serving-cert\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.889977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-snapshots\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsj84\" (UniqueName: \"kubernetes.io/projected/66c2abab-7f39-4d34-8571-ad883dde9532-kube-api-access-bsj84\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.890034 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87mw\" (UniqueName: \"kubernetes.io/projected/dd9fb448-f983-484e-85a4-afbd3efddd1d-kube-api-access-c87mw\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890110 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.890127 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.390106589 +0000 UTC m=+31.753774252 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : secret "router-metrics-certs-default" not found Nov 02 14:57:14.890190 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890182 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-service-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66c2abab-7f39-4d34-8571-ad883dde9532-serving-cert\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cgf\" (UniqueName: \"kubernetes.io/projected/f264a363-68bf-4fc1-87c6-824a8796d7ef-kube-api-access-45cgf\") pod \"kas-connection-checker-685895c7c9-7s4b6\" (UID: \"f264a363-68bf-4fc1-87c6-824a8796d7ef\") " pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.890411 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.890479 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.390460161 +0000 UTC m=+31.754127814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890413 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-config\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890627 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvwn\" (UniqueName: \"kubernetes.io/projected/f1684862-0c18-4f2f-87fa-811ca427d4a5-kube-api-access-mjvwn\") pod \"volume-data-source-validator-58c9f5f46d-k262f\" (UID: \"f1684862-0c18-4f2f-87fa-811ca427d4a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scq68\" (UniqueName: \"kubernetes.io/projected/f5cdff40-3fc5-423b-b14f-21172fab4f71-kube-api-access-scq68\") pod \"network-check-source-6f78d59f99-p8v6z\" (UID: \"f5cdff40-3fc5-423b-b14f-21172fab4f71\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.890692 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890733 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-snapshots\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f742m\" (UniqueName: \"kubernetes.io/projected/8e6e6852-e747-4869-aea2-5e0bbc44988f-kube-api-access-f742m\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890819 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/651ad3fc-4794-4a59-b41f-4fb01ed533d5-cert\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13c7b469-cf39-4d61-8f78-6fe03ba07dde-config\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/77135550-06c2-4f6b-8b7a-9ae0f81abb70-kube-api-access-nzpbn\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891199 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.891410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.890262 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891457 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6d3f56-0c8f-4631-aed3-7dedc802377a-config-volume\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891754 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-default-certificate\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e6e6852-e747-4869-aea2-5e0bbc44988f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891863 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.891887 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891892 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-tmp\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6d3f56-0c8f-4631-aed3-7dedc802377a-tmp-dir\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn6p\" (UniqueName: \"kubernetes.io/projected/8f6d3f56-0c8f-4631-aed3-7dedc802377a-kube-api-access-4hn6p\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.891995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892030 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-service-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892044 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg8g\" (UniqueName: \"kubernetes.io/projected/651ad3fc-4794-4a59-b41f-4fb01ed533d5-kube-api-access-ffg8g\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892104 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.892128 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.892141 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd9fb448-f983-484e-85a4-afbd3efddd1d-serving-cert\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.892300 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.892914 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.892379 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.392351096 +0000 UTC m=+31.756018735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:14.892914 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892848 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13c7b469-cf39-4d61-8f78-6fe03ba07dde-config\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.893019 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.892917 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.893073 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.893018 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:14.893073 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.893066 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.393050387 +0000 UTC m=+31.756718041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:14.893418 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.893395 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-tmp\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.893500 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.893400 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-stats-auth\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.893750 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.893730 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.393713752 +0000 UTC m=+31.757381404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:14.893853 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.893829 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.893921 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.893877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:14.893978 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.893950 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-config\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.893978 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.893963 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:14.894078 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.894001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.894078 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.894072 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.394045794 +0000 UTC m=+31.757713447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:14.894393 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.894374 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:14.894469 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.894406 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6d3f56-0c8f-4631-aed3-7dedc802377a-tmp-dir\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.894469 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.894431 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.394412694 +0000 UTC m=+31.758080354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:14.894601 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.894503 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:14.894601 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.894530 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:14.894601 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:14.894568 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:15.394556326 +0000 UTC m=+31.758223981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:14.894766 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.894643 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6d3f56-0c8f-4631-aed3-7dedc802377a-config-volume\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.894858 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.894835 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.895242 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.895218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13c7b469-cf39-4d61-8f78-6fe03ba07dde-serving-cert\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.895348 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.895312 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.895416 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.895371 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e6e6852-e747-4869-aea2-5e0bbc44988f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.895502 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.895479 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.895584 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.895497 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.896733 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.896355 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66c2abab-7f39-4d34-8571-ad883dde9532-serving-cert\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.896733 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.896404 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.896733 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.896696 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/651ad3fc-4794-4a59-b41f-4fb01ed533d5-cert\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.896858 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.896792 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.897307 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.897291 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd9fb448-f983-484e-85a4-afbd3efddd1d-serving-cert\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.897671 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.897654 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-default-certificate\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.937069 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.937046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:14.944333 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.944318 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:14.947689 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.947665 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvt6\" (UniqueName: \"kubernetes.io/projected/108f7147-992f-44ce-8cc4-dd7f8cd2521d-kube-api-access-mbvt6\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:14.947919 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.947895 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvwn\" (UniqueName: \"kubernetes.io/projected/f1684862-0c18-4f2f-87fa-811ca427d4a5-kube-api-access-mjvwn\") pod \"volume-data-source-validator-58c9f5f46d-k262f\" (UID: \"f1684862-0c18-4f2f-87fa-811ca427d4a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:14.948109 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.948092 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.948220 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.948201 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.948316 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.948299 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87mw\" (UniqueName: \"kubernetes.io/projected/dd9fb448-f983-484e-85a4-afbd3efddd1d-kube-api-access-c87mw\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:14.948639 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.948619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:14.948908 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.948892 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cgf\" (UniqueName: \"kubernetes.io/projected/f264a363-68bf-4fc1-87c6-824a8796d7ef-kube-api-access-45cgf\") pod \"kas-connection-checker-685895c7c9-7s4b6\" (UID: \"f264a363-68bf-4fc1-87c6-824a8796d7ef\") " pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:14.951035 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.951007 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsj84\" (UniqueName: \"kubernetes.io/projected/66c2abab-7f39-4d34-8571-ad883dde9532-kube-api-access-bsj84\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:14.951780 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.951764 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbc4v\" (UniqueName: \"kubernetes.io/projected/13c7b469-cf39-4d61-8f78-6fe03ba07dde-kube-api-access-rbc4v\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:14.958541 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.958221 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg8g\" (UniqueName: \"kubernetes.io/projected/651ad3fc-4794-4a59-b41f-4fb01ed533d5-kube-api-access-ffg8g\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:14.958606 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.958590 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/77135550-06c2-4f6b-8b7a-9ae0f81abb70-kube-api-access-nzpbn\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:14.959625 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.959599 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn6p\" (UniqueName: \"kubernetes.io/projected/8f6d3f56-0c8f-4631-aed3-7dedc802377a-kube-api-access-4hn6p\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:14.959986 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.959933 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scq68\" (UniqueName: \"kubernetes.io/projected/f5cdff40-3fc5-423b-b14f-21172fab4f71-kube-api-access-scq68\") pod \"network-check-source-6f78d59f99-p8v6z\" (UID: \"f5cdff40-3fc5-423b-b14f-21172fab4f71\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:14.960333 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.960304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f742m\" (UniqueName: \"kubernetes.io/projected/8e6e6852-e747-4869-aea2-5e0bbc44988f-kube-api-access-f742m\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:14.960512 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.960492 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:14.961579 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.961559 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:14.973592 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.973561 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:14.999479 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:14.999449 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:15.006112 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.005559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:15.016393 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.016367 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:15.022572 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.022551 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:15.030827 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.030008 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:15.046332 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.046311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:15.327711 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.327421 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-wfz5w"] Nov 02 14:57:15.329666 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.329227 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f"] Nov 02 14:57:15.330249 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.330225 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7hpmc"] Nov 02 14:57:15.332347 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.332321 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc4a9361_8679_4e65_83f4_d8d876321372.slice/crio-bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b WatchSource:0}: Error finding container bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b: Status 404 returned error can't find the container with id bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b Nov 02 14:57:15.333374 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.333299 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66a1a72_e152_44d7_aa2b_846e8a872078.slice/crio-c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f WatchSource:0}: Error finding container c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f: Status 404 returned error can't find the container with id c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f Nov 02 14:57:15.334263 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.334237 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88a6270_a93e_487d_a0b2_53a5b020c388.slice/crio-d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc WatchSource:0}: Error finding container d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc: Status 404 returned error can't find the container with id d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc Nov 02 14:57:15.372480 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.372453 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f"] Nov 02 14:57:15.381100 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.381071 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1684862_0c18_4f2f_87fa_811ca427d4a5.slice/crio-474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5 WatchSource:0}: Error finding container 474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5: Status 404 returned error can't find the container with id 474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5 Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.399834 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.399929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.399977 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.400004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.400032 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.400065 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.400125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.400163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400288 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400343 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400325906 +0000 UTC m=+32.763993559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400398 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400423 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400415713 +0000 UTC m=+32.764083364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400478 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400489 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:15.400967 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400537 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400510381 +0000 UTC m=+32.764178042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400606 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400595906 +0000 UTC m=+32.764263558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400663 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400672 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400690419 +0000 UTC m=+32.764358074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400751 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400778 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400769045 +0000 UTC m=+32.764436699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400832 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400861 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400851306 +0000 UTC m=+32.764518959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400915 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 14:57:15.401763 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:15.400942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:16.400933657 +0000 UTC m=+32.764601310 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : secret "router-metrics-certs-default" not found Nov 02 14:57:15.404778 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.404755 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-8vstz"] Nov 02 14:57:15.412004 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.411983 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9fb448_f983_484e_85a4_afbd3efddd1d.slice/crio-ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527 WatchSource:0}: Error finding container ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527: Status 404 returned error can't find the container with id ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527 Nov 02 14:57:15.429412 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.429391 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-r6gt7"] Nov 02 14:57:15.437660 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.437631 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c2abab_7f39_4d34_8571_ad883dde9532.slice/crio-7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3 WatchSource:0}: Error finding container 7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3: Status 404 returned error can't find the container with id 7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3 Nov 02 14:57:15.447176 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.447114 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-7s4b6"] Nov 02 14:57:15.449294 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.449271 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" event={"ID":"e66a1a72-e152-44d7-aa2b-846e8a872078","Type":"ContainerStarted","Data":"c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f"} Nov 02 14:57:15.450894 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.450871 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" event={"ID":"a88a6270-a93e-487d-a0b2-53a5b020c388","Type":"ContainerStarted","Data":"d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc"} Nov 02 14:57:15.452225 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.452204 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" event={"ID":"dd9fb448-f983-484e-85a4-afbd3efddd1d","Type":"ContainerStarted","Data":"ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527"} Nov 02 14:57:15.453073 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.453054 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" event={"ID":"f1684862-0c18-4f2f-87fa-811ca427d4a5","Type":"ContainerStarted","Data":"474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5"} Nov 02 14:57:15.454036 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.454018 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" event={"ID":"66c2abab-7f39-4d34-8571-ad883dde9532","Type":"ContainerStarted","Data":"7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3"} Nov 02 14:57:15.455237 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.454657 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf264a363_68bf_4fc1_87c6_824a8796d7ef.slice/crio-758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2 WatchSource:0}: Error finding container 758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2: Status 404 returned error can't find the container with id 758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2 Nov 02 14:57:15.455237 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.455140 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" event={"ID":"cc4a9361-8679-4e65-83f4-d8d876321372","Type":"ContainerStarted","Data":"bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b"} Nov 02 14:57:15.455852 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.455822 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z"] Nov 02 14:57:15.458865 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.458844 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql"] Nov 02 14:57:15.463771 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.463747 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5cdff40_3fc5_423b_b14f_21172fab4f71.slice/crio-3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a WatchSource:0}: Error finding container 3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a: Status 404 returned error can't find the container with id 3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a Nov 02 14:57:15.467753 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.467729 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c7b469_cf39_4d61_8f78_6fe03ba07dde.slice/crio-e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d WatchSource:0}: Error finding container e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d: Status 404 returned error can't find the container with id e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d Nov 02 14:57:15.484228 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:15.484206 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-r4vdl"] Nov 02 14:57:15.492668 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:15.492636 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651ad3fc_4794_4a59_b41f_4fb01ed533d5.slice/crio-68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e WatchSource:0}: Error finding container 68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e: Status 404 returned error can't find the container with id 68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e Nov 02 14:57:16.259049 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.259015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:16.259231 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.259015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:16.259231 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.259021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:16.313377 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.313340 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:57:16.313501 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.313440 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:57:16.313501 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.313481 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:57:16.313637 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.313500 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:57:16.313637 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.313528 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:57:16.407937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.407906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:16.407937 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.407939 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.407990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.408027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.408045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408053 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.408062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408069 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.408087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408108 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408093128 +0000 UTC m=+34.771760797 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408125 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408115907 +0000 UTC m=+34.771783546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408235 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.40822345 +0000 UTC m=+34.771891107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408238 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408255 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408313 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408295157 +0000 UTC m=+34.771962814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408327 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408382 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:16.408389 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408396 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408404 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408431 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408418717 +0000 UTC m=+34.772086371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.408400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408452 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408440544 +0000 UTC m=+34.772108184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408468 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408473 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408462756 +0000 UTC m=+34.772130395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:16.408889 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:16.408502 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:18.408492987 +0000 UTC m=+34.772160641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : secret "router-metrics-certs-default" not found Nov 02 14:57:16.459813 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.459771 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" event={"ID":"13c7b469-cf39-4d61-8f78-6fe03ba07dde","Type":"ContainerStarted","Data":"e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d"} Nov 02 14:57:16.462038 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.462009 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" event={"ID":"f5cdff40-3fc5-423b-b14f-21172fab4f71","Type":"ContainerStarted","Data":"3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a"} Nov 02 14:57:16.463530 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.463489 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4vdl" event={"ID":"651ad3fc-4794-4a59-b41f-4fb01ed533d5","Type":"ContainerStarted","Data":"68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e"} Nov 02 14:57:16.465666 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.465625 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" event={"ID":"f264a363-68bf-4fc1-87c6-824a8796d7ef","Type":"ContainerStarted","Data":"0d05363e2a7e8542e9be32305902cdd8a7b81e51c35f1b08c8655a10ff4826c9"} Nov 02 14:57:16.465757 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.465674 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" event={"ID":"f264a363-68bf-4fc1-87c6-824a8796d7ef","Type":"ContainerStarted","Data":"758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2"} Nov 02 14:57:16.474412 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.474379 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" event={"ID":"e66a1a72-e152-44d7-aa2b-846e8a872078","Type":"ContainerStarted","Data":"53712541b9e8c85777c8f088f3a83b7582bff4769a22e45032e444bf67cc5d75"} Nov 02 14:57:16.476752 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.476491 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" event={"ID":"a88a6270-a93e-487d-a0b2-53a5b020c388","Type":"ContainerStarted","Data":"18a9d384120af4f788bf176caa93bc59b44ff428e33d4807ce2107208b8ecaee"} Nov 02 14:57:16.650170 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.649999 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" podStartSLOduration=412.649979262 podStartE2EDuration="6m52.649979262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:16.647862649 +0000 UTC m=+33.011530311" watchObservedRunningTime="2026-11-02 14:57:16.649979262 +0000 UTC m=+33.013646927" Nov 02 14:57:16.871364 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:16.871304 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" podStartSLOduration=412.871285636 podStartE2EDuration="6m52.871285636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:16.760836372 +0000 UTC m=+33.124504036" watchObservedRunningTime="2026-11-02 14:57:16.871285636 +0000 UTC m=+33.234953292" Nov 02 14:57:17.116385 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.115557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:17.116385 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.115611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:17.116385 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:17.115704 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:57:17.116385 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:17.115771 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:49.115751645 +0000 UTC m=+65.479419298 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : secret "metrics-daemon-secret" not found Nov 02 14:57:17.134651 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.134589 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:17.179287 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.179050 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:17.217296 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.216737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:17.225994 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.225946 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:17.473775 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.473348 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:17.585286 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.585235 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" podStartSLOduration=413.585218831 podStartE2EDuration="6m53.585218831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:16.872149698 +0000 UTC m=+33.235817375" watchObservedRunningTime="2026-11-02 14:57:17.585218831 +0000 UTC m=+33.948886491" Nov 02 14:57:17.585876 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.585814 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88g4t"] Nov 02 14:57:17.611153 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:17.611108 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e70c22_8486_4834_8332_a8cf3e34d738.slice/crio-b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98 WatchSource:0}: Error finding container b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98: Status 404 returned error can't find the container with id b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98 Nov 02 14:57:17.875462 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:17.875281 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hk7f2"] Nov 02 14:57:17.887312 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:17.887252 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d40fd93_7004_40a5_a715_dcbac377dd94.slice/crio-0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561 WatchSource:0}: Error finding container 0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561: Status 404 returned error can't find the container with id 0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561 Nov 02 14:57:18.428738 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428643 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:18.428738 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:18.428738 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:18.428738 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428738 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:18.429040 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:18.429040 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:18.429040 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:18.429040 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.428857 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:18.429040 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429024 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:18.429275 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429081 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.429068176 +0000 UTC m=+38.792735827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:18.429570 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429546 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:18.429668 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429616 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:18.429668 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429665 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.429648907 +0000 UTC m=+38.793316560 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:18.429778 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429734 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.429723434 +0000 UTC m=+38.793391088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:18.429835 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429796 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:18.429835 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429805 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:18.429835 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429833 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.42982404 +0000 UTC m=+38.793491695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:18.429979 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429898 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:18.429979 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429919 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.429913429 +0000 UTC m=+38.793581068 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:18.429979 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429962 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:18.430118 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.429988 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.429980463 +0000 UTC m=+38.793648102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:18.430118 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.430025 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 14:57:18.430118 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.430044 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.430038722 +0000 UTC m=+38.793706374 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : secret "router-metrics-certs-default" not found Nov 02 14:57:18.430118 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.430076 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:18.430118 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:18.430094 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.430088727 +0000 UTC m=+38.793756366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:18.488171 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.488134 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" event={"ID":"f1684862-0c18-4f2f-87fa-811ca427d4a5","Type":"ContainerStarted","Data":"4472b7f2f5b632c0c8f327fe4d388d1020a4b4df3d9a3c8318422375d02a7847"} Nov 02 14:57:18.490690 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.490615 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hk7f2" event={"ID":"4d40fd93-7004-40a5-a715-dcbac377dd94","Type":"ContainerStarted","Data":"0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561"} Nov 02 14:57:18.492438 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.492346 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88g4t" event={"ID":"a3e70c22-8486-4834-8332-a8cf3e34d738","Type":"ContainerStarted","Data":"b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98"} Nov 02 14:57:18.606187 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:18.605353 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" podStartSLOduration=377.371474607 podStartE2EDuration="6m19.605338369s" totalImagesPullingTime="2.233863762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:18.603942389 +0000 UTC m=+34.967610100" watchObservedRunningTime="2026-11-02 14:57:18.605338369 +0000 UTC m=+34.969006029" Nov 02 14:57:20.518273 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.517480 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" event={"ID":"66c2abab-7f39-4d34-8571-ad883dde9532","Type":"ContainerStarted","Data":"bc03f3ffa4100f6d7374261a489ab8bb128c4730fd6aa503c893f239b1669d85"} Nov 02 14:57:20.523764 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.523735 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" event={"ID":"cc4a9361-8679-4e65-83f4-d8d876321372","Type":"ContainerStarted","Data":"971f2ed6513fdfd366b7ff40b6ba7a9302f80dc8cf6a0a1bd61b42b6240bc9fd"} Nov 02 14:57:20.526610 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.526588 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/0.log" Nov 02 14:57:20.526727 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.526626 2580 generic.go:356] "Generic (PLEG): container finished" podID="dd9fb448-f983-484e-85a4-afbd3efddd1d" containerID="7030365d2ef99e62260a8fef65f3102c555e749d6b9f14e12ee5192dfd19fefa" exitCode=255 Nov 02 14:57:20.526727 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.526669 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" event={"ID":"dd9fb448-f983-484e-85a4-afbd3efddd1d","Type":"ContainerDied","Data":"7030365d2ef99e62260a8fef65f3102c555e749d6b9f14e12ee5192dfd19fefa"} Nov 02 14:57:20.526913 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.526899 2580 scope.go:122] "RemoveContainer" containerID="7030365d2ef99e62260a8fef65f3102c555e749d6b9f14e12ee5192dfd19fefa" Nov 02 14:57:20.744773 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.744371 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" podStartSLOduration=370.902981107 podStartE2EDuration="6m15.744351235s" totalImagesPullingTime="4.841370128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:20.631458752 +0000 UTC m=+36.995126425" watchObservedRunningTime="2026-11-02 14:57:20.744351235 +0000 UTC m=+37.108018899" Nov 02 14:57:20.854268 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:20.854204 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" podStartSLOduration=376.97810404 podStartE2EDuration="6m21.854185315s" totalImagesPullingTime="4.876081275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:20.853042859 +0000 UTC m=+37.216710519" watchObservedRunningTime="2026-11-02 14:57:20.854185315 +0000 UTC m=+37.217852979" Nov 02 14:57:21.540645 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.540551 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/1.log" Nov 02 14:57:21.541360 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.541338 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/0.log" Nov 02 14:57:21.541481 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.541382 2580 generic.go:356] "Generic (PLEG): container finished" podID="dd9fb448-f983-484e-85a4-afbd3efddd1d" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" exitCode=255 Nov 02 14:57:21.541655 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.541616 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" event={"ID":"dd9fb448-f983-484e-85a4-afbd3efddd1d","Type":"ContainerDied","Data":"fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2"} Nov 02 14:57:21.541753 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.541689 2580 scope.go:122] "RemoveContainer" containerID="7030365d2ef99e62260a8fef65f3102c555e749d6b9f14e12ee5192dfd19fefa" Nov 02 14:57:21.542136 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.541825 2580 scope.go:122] "RemoveContainer" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" Nov 02 14:57:21.542136 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:21.542125 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8vstz_openshift-console-operator(dd9fb448-f983-484e-85a4-afbd3efddd1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" Nov 02 14:57:21.544782 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.544707 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" event={"ID":"13c7b469-cf39-4d61-8f78-6fe03ba07dde","Type":"ContainerStarted","Data":"3b0de216106be53058fba5e77f2202ebcb1578d91a3b6456f314ba5a3da74245"} Nov 02 14:57:21.548801 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.548319 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-r4vdl" event={"ID":"651ad3fc-4794-4a59-b41f-4fb01ed533d5","Type":"ContainerStarted","Data":"a167d1cb12bc75eea958997407af278fab59c3887614647aa16142251464771b"} Nov 02 14:57:21.768453 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.768341 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" podStartSLOduration=376.069956339 podStartE2EDuration="6m21.768320618s" totalImagesPullingTime="5.698364279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:21.767354281 +0000 UTC m=+38.131021968" watchObservedRunningTime="2026-11-02 14:57:21.768320618 +0000 UTC m=+38.131988282" Nov 02 14:57:21.881951 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:21.881890 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-r4vdl" podStartSLOduration=1.9464236910000001 podStartE2EDuration="7.881869355s" totalImagesPullingTime="5.935445664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:21.880806615 +0000 UTC m=+38.244474304" watchObservedRunningTime="2026-11-02 14:57:21.881869355 +0000 UTC m=+38.245537017" Nov 02 14:57:22.050252 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.050176 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq"] Nov 02 14:57:22.054971 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.054940 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:22.064196 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.064171 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8"] Nov 02 14:57:22.067558 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.067495 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:22.106025 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.105474 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:22.108421 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.108387 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq"] Nov 02 14:57:22.108569 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.108540 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 14:57:22.108814 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.108726 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kbp57\"" Nov 02 14:57:22.165707 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.165673 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8"] Nov 02 14:57:22.175284 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.175257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r27x\" (UniqueName: \"kubernetes.io/projected/0fe317c6-3f53-41ef-8448-1c453427bf9f-kube-api-access-4r27x\") pod \"migrator-66df5d496b-bk2l8\" (UID: \"0fe317c6-3f53-41ef-8448-1c453427bf9f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:22.175483 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.175466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsflx\" (UniqueName: \"kubernetes.io/projected/a0698158-f320-40dc-8b65-bd56c57e4ac7-kube-api-access-xsflx\") pod \"migrator-66df5d496b-zgzbq\" (UID: \"a0698158-f320-40dc-8b65-bd56c57e4ac7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:22.277147 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.276402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r27x\" (UniqueName: \"kubernetes.io/projected/0fe317c6-3f53-41ef-8448-1c453427bf9f-kube-api-access-4r27x\") pod \"migrator-66df5d496b-bk2l8\" (UID: \"0fe317c6-3f53-41ef-8448-1c453427bf9f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:22.277147 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.276604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsflx\" (UniqueName: \"kubernetes.io/projected/a0698158-f320-40dc-8b65-bd56c57e4ac7-kube-api-access-xsflx\") pod \"migrator-66df5d496b-zgzbq\" (UID: \"a0698158-f320-40dc-8b65-bd56c57e4ac7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:22.287451 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.287423 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r2l82"] Nov 02 14:57:22.292542 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.292503 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.337439 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.337407 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsflx\" (UniqueName: \"kubernetes.io/projected/a0698158-f320-40dc-8b65-bd56c57e4ac7-kube-api-access-xsflx\") pod \"migrator-66df5d496b-zgzbq\" (UID: \"a0698158-f320-40dc-8b65-bd56c57e4ac7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:22.341295 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.341265 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:57:22.341418 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.341366 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:57:22.341679 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.341640 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2bv5\"" Nov 02 14:57:22.344061 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.344031 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r27x\" (UniqueName: \"kubernetes.io/projected/0fe317c6-3f53-41ef-8448-1c453427bf9f-kube-api-access-4r27x\") pod \"migrator-66df5d496b-bk2l8\" (UID: \"0fe317c6-3f53-41ef-8448-1c453427bf9f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:22.346635 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.346612 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r2l82"] Nov 02 14:57:22.370097 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.370065 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:22.378124 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.377711 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-crio-socket\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.378124 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.377752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-data-volume\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.378124 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.377848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.378124 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.377896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqs7c\" (UniqueName: \"kubernetes.io/projected/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-api-access-bqs7c\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.378124 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.377924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.380724 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.380701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:22.479383 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:22.479510 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs7c\" (UniqueName: \"kubernetes.io/projected/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-api-access-bqs7c\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.479510 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479421 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.479510 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:22.479700 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:22.479700 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:22.479700 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-crio-socket\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.479700 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479694 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-data-volume\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.479899 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:22.479899 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479782 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:22.479899 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479810 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:22.479899 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.479899 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.479862 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:22.480148 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.480004 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:22.480148 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.480068 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.480051137 +0000 UTC m=+46.843718790 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:22.480678 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.480657 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:22.480817 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.480804 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.480788474 +0000 UTC m=+46.844456128 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:22.481135 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481115 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:22.481201 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481167 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.981153175 +0000 UTC m=+39.344820833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:22.481238 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481227 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 02 14:57:22.481266 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481260 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.48124995 +0000 UTC m=+46.844917603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : secret "router-metrics-certs-default" not found Nov 02 14:57:22.481320 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481308 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:22.481352 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481342 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.481332996 +0000 UTC m=+46.845000650 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:22.481403 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481392 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:22.481440 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481428 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.481418763 +0000 UTC m=+46.845086416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:22.481500 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.481484 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-crio-socket\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.481646 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481626 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:22.481706 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481650 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:22.481706 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481651 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.481613642 +0000 UTC m=+46.845281298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:22.481787 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481707 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.481688408 +0000 UTC m=+46.845356062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:22.481787 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481727 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:22.481787 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481746 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:22.481894 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481807 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.481789967 +0000 UTC m=+46.845457622 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:22.481894 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481814 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:57:22.481894 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.481878 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:22.981850491 +0000 UTC m=+39.345518145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : configmap "kube-rbac-proxy" not found Nov 02 14:57:22.482143 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.482125 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-data-volume\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.517697 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.517656 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2nf2p-5rnb4"] Nov 02 14:57:22.522830 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.522806 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.545778 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.545716 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs7c\" (UniqueName: \"kubernetes.io/projected/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-api-access-bqs7c\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.554323 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.554270 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/1.log" Nov 02 14:57:22.555337 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.554644 2580 scope.go:122] "RemoveContainer" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" Nov 02 14:57:22.555337 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.554833 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8vstz_openshift-console-operator(dd9fb448-f983-484e-85a4-afbd3efddd1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" Nov 02 14:57:22.576108 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.576076 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2nf2p-5rnb4"] Nov 02 14:57:22.681445 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.681411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.681621 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.681484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.681801 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.681769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.681909 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.681858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.759385 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.759314 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq"] Nov 02 14:57:22.778562 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:22.778533 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0698158_f320_40dc_8b65_bd56c57e4ac7.slice/crio-6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058 WatchSource:0}: Error finding container 6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058: Status 404 returned error can't find the container with id 6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058 Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.782888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.782955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.783009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.783060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.783920 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.784132 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.784183 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.784153 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.803680 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.803615 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8"] Nov 02 14:57:22.810798 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:22.810512 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe317c6_3f53_41ef_8448_1c453427bf9f.slice/crio-57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04 WatchSource:0}: Error finding container 57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04: Status 404 returned error can't find the container with id 57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04 Nov 02 14:57:22.846430 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.846398 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:22.985218 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.985184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.985398 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:22.985253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:22.985483 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.985467 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:22.985578 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.985563 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:23.98553201 +0000 UTC m=+40.349199667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:22.985654 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.985633 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:57:22.985708 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:22.985673 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:23.985661747 +0000 UTC m=+40.349329404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : configmap "kube-rbac-proxy" not found Nov 02 14:57:23.140173 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.140121 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:23.520090 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.520004 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2nf2p-5rnb4"] Nov 02 14:57:23.527387 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:23.527356 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ef143c_9312_4bc0_8180_7864b729d2b2.slice/crio-d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22 WatchSource:0}: Error finding container d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22: Status 404 returned error can't find the container with id d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22 Nov 02 14:57:23.563562 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.563535 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" event={"ID":"0fe317c6-3f53-41ef-8448-1c453427bf9f","Type":"ContainerStarted","Data":"57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04"} Nov 02 14:57:23.565271 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.565247 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" event={"ID":"f5cdff40-3fc5-423b-b14f-21172fab4f71","Type":"ContainerStarted","Data":"96f6d01606b67579b96de0d50a1273410ac6994fa656b50c2e70533f7ee70655"} Nov 02 14:57:23.566566 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.566542 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" event={"ID":"60ef143c-9312-4bc0-8180-7864b729d2b2","Type":"ContainerStarted","Data":"d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22"} Nov 02 14:57:23.567753 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.567730 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" event={"ID":"a0698158-f320-40dc-8b65-bd56c57e4ac7","Type":"ContainerStarted","Data":"6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058"} Nov 02 14:57:23.674600 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.674554 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" podStartSLOduration=368.42758105 podStartE2EDuration="6m15.674537627s" totalImagesPullingTime="7.246956577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:23.673643378 +0000 UTC m=+40.037311255" watchObservedRunningTime="2026-11-02 14:57:23.674537627 +0000 UTC m=+40.038205286" Nov 02 14:57:23.998670 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.997708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:23.998670 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:23.997790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:23.998670 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:23.997970 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:23.998670 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:23.998064 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:25.998045462 +0000 UTC m=+42.361713116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:23.998670 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:23.998545 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:57:23.998670 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:23.998601 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:25.998584387 +0000 UTC m=+42.362252064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : configmap "kube-rbac-proxy" not found Nov 02 14:57:24.001978 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.001951 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jpb5m"] Nov 02 14:57:24.005872 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.005845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.055399 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.055301 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 14:57:24.055399 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.055359 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 14:57:24.055697 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.055445 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 14:57:24.055756 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.055705 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 14:57:24.055812 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.055784 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xhs42\"" Nov 02 14:57:24.081399 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.081364 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jpb5m"] Nov 02 14:57:24.098352 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.098309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-key\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.098511 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.098486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-cabundle\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.098615 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.098563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4gb\" (UniqueName: \"kubernetes.io/projected/e28966c5-8bda-4f1b-a66a-ba3493523a38-kube-api-access-mz4gb\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.199997 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.199937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-key\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.200181 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.200064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-cabundle\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.200181 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.200121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4gb\" (UniqueName: \"kubernetes.io/projected/e28966c5-8bda-4f1b-a66a-ba3493523a38-kube-api-access-mz4gb\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.201708 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.201635 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-cabundle\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.204063 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.204040 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-key\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.258547 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.258428 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4gb\" (UniqueName: \"kubernetes.io/projected/e28966c5-8bda-4f1b-a66a-ba3493523a38-kube-api-access-mz4gb\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.322824 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.322792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:24.575053 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.574975 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hk7f2" event={"ID":"4d40fd93-7004-40a5-a715-dcbac377dd94","Type":"ContainerStarted","Data":"f65af2a7128c3aa0777f573d12ac3ff054ab8c344d74c708886a9148edf1b52e"} Nov 02 14:57:24.575635 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.575612 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:24.578542 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.578420 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" event={"ID":"60ef143c-9312-4bc0-8180-7864b729d2b2","Type":"ContainerStarted","Data":"edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a"} Nov 02 14:57:24.689251 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.689193 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hk7f2" podStartSLOduration=34.937214069 podStartE2EDuration="40.689176086s" totalImagesPullingTime="5.751962017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:24.687688742 +0000 UTC m=+41.051356405" watchObservedRunningTime="2026-11-02 14:57:24.689176086 +0000 UTC m=+41.052843748" Nov 02 14:57:24.697410 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.697381 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-jpb5m"] Nov 02 14:57:24.704344 ip-10-0-130-229 kubenswrapper[2580]: W1102 14:57:24.704313 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28966c5_8bda_4f1b_a66a_ba3493523a38.slice/crio-15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e WatchSource:0}: Error finding container 15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e: Status 404 returned error can't find the container with id 15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e Nov 02 14:57:24.801599 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:24.801561 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" podStartSLOduration=2.801543609 podStartE2EDuration="2.801543609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:24.800907322 +0000 UTC m=+41.164575006" watchObservedRunningTime="2026-11-02 14:57:24.801543609 +0000 UTC m=+41.165211267" Nov 02 14:57:25.000652 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.000617 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:25.000827 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.000682 2580 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:25.001137 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.001109 2580 scope.go:122] "RemoveContainer" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" Nov 02 14:57:25.001364 ip-10-0-130-229 kubenswrapper[2580]: E1102 14:57:25.001335 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8vstz_openshift-console-operator(dd9fb448-f983-484e-85a4-afbd3efddd1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" Nov 02 14:57:25.582471 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.582440 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" event={"ID":"e28966c5-8bda-4f1b-a66a-ba3493523a38","Type":"ContainerStarted","Data":"14f882d5cec664071ec990bbc1a4491b800d68a3d0f3ea73c0b1fa2bcf435866"} Nov 02 14:57:25.582841 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.582499 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" event={"ID":"e28966c5-8bda-4f1b-a66a-ba3493523a38","Type":"ContainerStarted","Data":"15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e"} Nov 02 14:57:25.584160 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.584136 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" event={"ID":"0fe317c6-3f53-41ef-8448-1c453427bf9f","Type":"ContainerStarted","Data":"ab3a0ed33727b4e9e24fc20542d8af0a702f633def93905d1fd11b4689dadbb7"} Nov 02 14:57:25.585795 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.585738 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" event={"ID":"a0698158-f320-40dc-8b65-bd56c57e4ac7","Type":"ContainerStarted","Data":"d594cc465e63d145383912e2576b2c670d548cab946f187f91f8ca22c78c1a40"} Nov 02 14:57:25.755901 ip-10-0-130-229 kubenswrapper[2580]: I1102 14:57:25.752738 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" podStartSLOduration=2.75272167 podStartE2EDuration="2.75272167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:25.75081626 +0000 UTC m=+42.114483921" watchObservedRunningTime="2026-11-02 14:57:25.75272167 +0000 UTC m=+42.116389332" Nov 02 14:57:25.771653 ip-10-0-130-229 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 14:57:25.785187 ip-10-0-130-229 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 14:57:25.785365 ip-10-0-130-229 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 14:57:25.785556 ip-10-0-130-229 systemd[1]: kubelet.service: Consumed 3.200s CPU time. Nov 02 14:57:25.805015 ip-10-0-130-229 systemd[1]: Starting Kubernetes Kubelet... Nov 02 14:57:25.865379 ip-10-0-130-229 kubenswrapper[5091]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:25.865379 ip-10-0-130-229 kubenswrapper[5091]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 14:57:25.865379 ip-10-0-130-229 kubenswrapper[5091]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:25.865379 ip-10-0-130-229 kubenswrapper[5091]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 14:57:25.867370 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867299 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.867370 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867370 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867374 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867377 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867380 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867383 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867386 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867388 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867392 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867394 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867397 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867400 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867404 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867408 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867411 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867414 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867417 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867420 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867423 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867426 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.867441 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867428 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867431 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867433 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867436 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867438 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867441 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867444 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867447 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867460 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867464 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867466 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867469 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867471 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867474 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867476 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867479 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867482 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867484 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867487 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867489 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.867919 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867492 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867494 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867497 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867499 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867502 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867504 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867507 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867510 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867538 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867543 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867547 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867550 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867553 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867556 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867559 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867562 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867564 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867567 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867570 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867572 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.868402 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867576 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867585 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867588 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867591 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867594 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867597 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867599 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867602 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867605 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867607 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867610 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867613 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867615 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867618 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867620 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867623 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867625 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867628 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867631 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867633 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.868890 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867636 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867638 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867641 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867643 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867646 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867651 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867654 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867657 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867660 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867662 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867665 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867667 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867670 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867673 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867676 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867679 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867681 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867684 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867687 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.869374 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867689 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867692 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867695 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867697 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867700 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867703 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.867706 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868529 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868537 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868541 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868544 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868547 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868550 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868553 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868555 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868558 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868561 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868563 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868569 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868572 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.869841 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868575 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868577 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868580 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868582 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868585 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868588 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868590 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868593 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868601 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868604 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868607 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868610 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868612 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868615 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868618 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868620 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868623 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868625 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868628 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.870343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868630 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868633 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868636 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868639 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868641 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868644 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868646 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868649 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868652 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868654 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868657 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868660 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868663 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868666 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868668 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868671 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868674 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868676 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868679 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.870814 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868681 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868684 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868687 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868699 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868703 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868707 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868711 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868714 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868718 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868721 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868723 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868726 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868734 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868737 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868740 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868742 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868745 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868748 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868750 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.871312 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868753 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868755 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868758 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868761 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868763 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868768 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868771 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868773 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868776 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868778 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868781 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868784 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868786 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868789 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868791 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868794 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868797 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868808 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868811 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868813 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.871952 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868816 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868818 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868821 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868823 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868826 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868828 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868831 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868833 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868836 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868838 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868841 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868843 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868846 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868849 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868851 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.868854 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869431 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869439 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869443 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869446 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.872435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869450 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869452 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869455 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869458 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869461 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869463 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869466 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869468 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869471 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869479 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869482 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869485 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869487 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869490 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869492 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869495 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869497 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869499 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869502 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.872942 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869505 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869507 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869509 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869525 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869528 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869531 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869534 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869537 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869539 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869542 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869546 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869549 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869553 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869556 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869559 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869562 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869564 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869567 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869570 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.873459 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869572 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869575 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869577 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869580 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869588 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869591 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869594 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869597 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869600 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869602 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869605 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869607 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869610 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869612 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869615 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869617 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869620 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869623 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869625 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869628 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.873931 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869630 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869632 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869635 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869640 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869643 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869645 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869648 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869651 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869653 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869656 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869659 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869661 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869664 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869667 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869669 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869672 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869675 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869683 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869685 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.874690 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869688 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869691 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869693 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869696 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869698 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869701 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869703 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869706 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869709 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869711 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869714 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869721 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869724 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869727 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869729 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869732 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869738 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869742 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869745 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.875291 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869748 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869752 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869754 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869757 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869759 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.869762 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869851 5091 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869858 5091 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869872 5091 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869877 5091 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869881 5091 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869885 5091 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869895 5091 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869899 5091 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869902 5091 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869905 5091 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869909 5091 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869912 5091 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869915 5091 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869918 5091 flags.go:64] FLAG: --cgroup-root="" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869921 5091 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869925 5091 flags.go:64] FLAG: --client-ca-file="" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869928 5091 flags.go:64] FLAG: --cloud-provider="external" Nov 02 14:57:25.875776 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869931 5091 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869937 5091 flags.go:64] FLAG: --cluster-domain="" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869940 5091 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869943 5091 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869946 5091 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869950 5091 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869955 5091 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869958 5091 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869961 5091 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869965 5091 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869968 5091 flags.go:64] FLAG: --contention-profiling="false" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869971 5091 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869974 5091 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869977 5091 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869980 5091 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869984 5091 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869987 5091 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869990 5091 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869993 5091 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869996 5091 flags.go:64] FLAG: --enable-server="true" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.869999 5091 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870009 5091 flags.go:64] FLAG: --event-burst="100" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870020 5091 flags.go:64] FLAG: --event-qps="50" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870023 5091 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 14:57:25.876330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870026 5091 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870029 5091 flags.go:64] FLAG: --eviction-hard="" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870033 5091 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870037 5091 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870040 5091 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870043 5091 flags.go:64] FLAG: --eviction-soft="" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870046 5091 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870049 5091 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870052 5091 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870055 5091 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870058 5091 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870060 5091 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870063 5091 flags.go:64] FLAG: --feature-gates="" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870067 5091 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870070 5091 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870074 5091 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870077 5091 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870080 5091 flags.go:64] FLAG: --healthz-port="10248" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870083 5091 flags.go:64] FLAG: --help="false" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870085 5091 flags.go:64] FLAG: --hostname-override="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870089 5091 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870092 5091 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870095 5091 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 14:57:25.876904 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870098 5091 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870101 5091 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870104 5091 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870107 5091 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870110 5091 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870113 5091 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870116 5091 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870119 5091 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870127 5091 flags.go:64] FLAG: --kube-reserved="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870131 5091 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870134 5091 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870137 5091 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870141 5091 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870144 5091 flags.go:64] FLAG: --lock-file="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870147 5091 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870150 5091 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870153 5091 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870159 5091 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870161 5091 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870165 5091 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870168 5091 flags.go:64] FLAG: --logging-format="text" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870171 5091 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870174 5091 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870177 5091 flags.go:64] FLAG: --manifest-url="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870179 5091 flags.go:64] FLAG: --manifest-url-header="" Nov 02 14:57:25.877451 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870184 5091 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870187 5091 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870190 5091 flags.go:64] FLAG: --max-pods="110" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870193 5091 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870196 5091 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870200 5091 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870203 5091 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870205 5091 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870208 5091 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870211 5091 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870218 5091 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870221 5091 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870224 5091 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870227 5091 flags.go:64] FLAG: --pod-cidr="" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870230 5091 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870233 5091 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870246 5091 flags.go:64] FLAG: --pods-per-core="0" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870249 5091 flags.go:64] FLAG: --port="10250" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870252 5091 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870256 5091 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0a56a95937141b516" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870260 5091 flags.go:64] FLAG: --qos-reserved="" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870263 5091 flags.go:64] FLAG: --read-only-port="10255" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870266 5091 flags.go:64] FLAG: --register-node="true" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870269 5091 flags.go:64] FLAG: --register-with-taints="" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870272 5091 flags.go:64] FLAG: --registry-burst="10" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870275 5091 flags.go:64] FLAG: --registry-qps="5" Nov 02 14:57:25.878058 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870278 5091 flags.go:64] FLAG: --reserved-cpus="" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870281 5091 flags.go:64] FLAG: --reserved-memory="" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870285 5091 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870289 5091 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870293 5091 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870296 5091 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870299 5091 flags.go:64] FLAG: --runonce="false" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870301 5091 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870305 5091 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870308 5091 flags.go:64] FLAG: --seccomp-default="false" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870310 5091 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870313 5091 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870317 5091 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870320 5091 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870323 5091 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870326 5091 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870329 5091 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870331 5091 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870334 5091 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870337 5091 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870340 5091 flags.go:64] FLAG: --system-cgroups="" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870343 5091 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870349 5091 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870355 5091 flags.go:64] FLAG: --tls-cert-file="" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870359 5091 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 14:57:25.878714 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870368 5091 flags.go:64] FLAG: --tls-min-version="" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870371 5091 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870374 5091 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870377 5091 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870380 5091 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870383 5091 flags.go:64] FLAG: --v="2" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870388 5091 flags.go:64] FLAG: --version="false" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870392 5091 flags.go:64] FLAG: --vmodule="" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870396 5091 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870400 5091 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870590 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870596 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870600 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870603 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870606 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870609 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870612 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870615 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870618 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870621 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870623 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.879306 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870626 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870628 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870631 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870633 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870636 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870639 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870641 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870644 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870647 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870649 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870658 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870663 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870665 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870668 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870671 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870673 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870676 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870678 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870681 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870683 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.879856 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870686 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870690 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870693 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870696 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870698 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870701 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870703 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870706 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870709 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870711 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870714 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870716 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870720 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870724 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870727 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870729 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870732 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870735 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870738 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.880361 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870740 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870743 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870745 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870748 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870771 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870775 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870778 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870781 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870784 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870787 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870790 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870792 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870795 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870797 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870801 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870804 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870806 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870810 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870814 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.880915 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870818 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870821 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870824 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870827 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870829 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870832 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870835 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870837 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870840 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870842 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870845 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870848 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870851 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870853 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870856 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870858 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870861 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870865 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870873 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870875 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.881424 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870878 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870881 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870883 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870886 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870888 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870891 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870893 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870898 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870900 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870903 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870906 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870908 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870911 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870914 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870917 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870920 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.881973 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870922 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.870947 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870992 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870996 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.870999 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871002 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871005 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871007 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871010 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871012 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871015 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871018 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871020 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871024 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871027 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871039 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.882435 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871042 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871045 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871047 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871050 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871052 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871056 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871060 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871062 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871066 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871076 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871080 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871082 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871085 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871095 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871098 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871100 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871103 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871106 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871108 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871111 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.882850 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871113 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871116 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871119 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871121 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871124 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871127 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871129 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871132 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871134 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871139 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871143 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871147 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871155 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871158 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871161 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871164 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871167 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871170 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871173 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.883343 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871175 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871179 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871182 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871185 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871187 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871190 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871193 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871195 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871198 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871201 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871203 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871206 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871208 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871211 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871213 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871216 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871219 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871221 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871224 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871227 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.884091 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871229 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871232 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871236 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871239 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871242 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871245 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871253 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871256 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871258 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871261 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871263 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871266 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871269 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871272 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871275 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871278 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871281 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871283 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871286 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.884730 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871289 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871291 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871294 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871296 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871299 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871302 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871304 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871306 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871309 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871312 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871315 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871318 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871321 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.871323 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.876245 5091 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.876262 5091 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.876285 5091 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.876290 5091 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876351 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876356 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.885190 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876360 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876363 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876366 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876369 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876372 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876375 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876378 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876380 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876383 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876385 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876388 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876395 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876399 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876401 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876404 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876407 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876410 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876413 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876416 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876418 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.885691 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876422 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876424 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876427 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876430 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876432 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876435 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876438 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876440 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876443 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876446 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876448 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876452 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876454 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876457 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876459 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876462 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876464 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876467 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876469 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.886165 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876472 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876475 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876478 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876480 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876483 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876487 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876490 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876492 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876495 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876498 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876501 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876504 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876507 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876509 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876526 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876532 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876537 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876540 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876544 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.886641 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876546 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876549 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876552 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876555 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876558 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876560 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876563 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876566 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876568 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876571 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876573 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876576 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876578 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876581 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876583 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876586 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876588 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876591 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876594 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876598 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.887109 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876601 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876603 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876606 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876609 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876612 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876614 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876617 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876619 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876622 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876624 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876627 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876630 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876634 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876637 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876640 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876642 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876646 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876649 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876652 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.887616 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876654 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876657 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876660 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876663 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876665 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876668 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876670 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.876697 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876836 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876841 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876845 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876848 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876851 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876855 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876857 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876860 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.888142 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876863 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876865 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876868 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876870 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876873 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876875 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876878 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876881 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876884 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876886 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876888 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876891 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876894 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876896 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876899 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876902 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876904 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876907 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876910 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.888742 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876912 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876915 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876918 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876921 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876923 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876926 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876929 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876931 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876934 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876936 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876939 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876942 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876945 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876947 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876950 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876952 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876955 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876957 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876960 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876963 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.889263 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876965 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876968 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876971 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876974 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876976 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876979 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876981 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876984 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876986 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876988 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876991 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876993 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876996 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.876998 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877001 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877003 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877006 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877008 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877011 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877013 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.890036 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877016 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877018 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877020 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877023 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877026 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877029 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877033 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877036 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877039 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877042 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877045 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877047 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877050 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877052 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877054 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877057 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877060 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877064 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.890625 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877067 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877069 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877072 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877074 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877077 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877079 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877082 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877084 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877087 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877090 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877092 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877094 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877097 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877100 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877102 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877105 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877107 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877110 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877113 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.891066 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877116 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877118 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.877140 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877171 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877174 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877178 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877181 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877183 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877186 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877188 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877191 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877194 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877196 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877199 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877201 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877204 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 14:57:25.891621 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877206 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877209 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877211 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877214 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877217 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877220 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877222 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877225 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877227 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877230 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877232 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877235 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877237 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877240 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877243 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877245 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877248 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877250 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877254 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877257 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 14:57:25.892033 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877259 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877262 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877265 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877268 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877270 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877273 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877276 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877278 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877281 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877283 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877286 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877288 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877291 5091 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877293 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877296 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877298 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877301 5091 feature_gate.go:386] unrecognized feature gate: Example Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877303 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877306 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877308 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 14:57:25.892537 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877311 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877313 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877316 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877318 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877321 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877323 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877326 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877328 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877332 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877336 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877339 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877342 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877344 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877347 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877349 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877352 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877354 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877357 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877360 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 14:57:25.893436 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877363 5091 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877366 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877369 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877372 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877375 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877379 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877382 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877385 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877387 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877390 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877392 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877395 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877397 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877400 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877402 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877405 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877407 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877410 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877412 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 14:57:25.894148 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877415 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877418 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877420 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877423 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877426 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877429 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877432 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877435 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877437 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877440 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877443 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877445 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877447 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877450 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:25.877453 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.877531 5091 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.878467 5091 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.878538 5091 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.878724 5091 server.go:1014] "Starting client certificate rotation" Nov 02 14:57:25.894619 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.878862 5091 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.878912 5091 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 14:51:44 +0000 UTC" deadline="2028-04-29 05:18:51.572014135 +0000 UTC" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.878967 5091 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13046h21m25.693048674s" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.879175 5091 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.880284 5091 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.881600 5091 log.go:25] "Validated CRI v1 runtime API" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.883481 5091 log.go:25] "Validated CRI v1 image API" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.884201 5091 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 14:57:25.895080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.886281 5091 fs.go:135] Filesystem UUIDs: map[3a3875af-2fe1-4231-a91f-6b9ecf150048:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 83c89c56-23d0-4337-9896-8d171a9f9c2f:/dev/nvme0n1p4] Nov 02 14:57:25.895319 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.886300 5091 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0991e750-1337-47d8-a151-1c2ed20797f6/volumes/kubernetes.io~projected/kube-api-access-5p8wb:{mountpoint:/var/lib/kubelet/pods/0991e750-1337-47d8-a151-1c2ed20797f6/volumes/kubernetes.io~projected/kube-api-access-5p8wb major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29/volumes/kubernetes.io~projected/kube-api-access-ffq26:{mountpoint:/var/lib/kubelet/pods/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29/volumes/kubernetes.io~projected/kube-api-access-ffq26 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fe317c6-3f53-41ef-8448-1c453427bf9f/volumes/kubernetes.io~projected/kube-api-access-4r27x:{mountpoint:/var/lib/kubelet/pods/0fe317c6-3f53-41ef-8448-1c453427bf9f/volumes/kubernetes.io~projected/kube-api-access-4r27x major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~projected/kube-api-access-mbvt6:{mountpoint:/var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~projected/kube-api-access-mbvt6 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~secret/stats-auth major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13c7b469-cf39-4d61-8f78-6fe03ba07dde/volumes/kubernetes.io~projected/kube-api-access-rbc4v:{mountpoint:/var/lib/kubelet/pods/13c7b469-cf39-4d61-8f78-6fe03ba07dde/volumes/kubernetes.io~projected/kube-api-access-rbc4v major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13c7b469-cf39-4d61-8f78-6fe03ba07dde/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/13c7b469-cf39-4d61-8f78-6fe03ba07dde/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~projected/kube-api-access-6dc7f:{mountpoint:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~projected/kube-api-access-6dc7f major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~projected/kube-api-access-dsnm7:{mountpoint:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~projected/kube-api-access-dsnm7 major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~projected/kube-api-access-lgbgt:{mountpoint:/var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~projected/kube-api-access-lgbgt major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a16a23a-e4f9-4fd7-a900-28027f31c7de/volumes/kubernetes.io~projected/kube-api-access-26d2p:{mountpoint:/var/lib/kubelet/pods/4a16a23a-e4f9-4fd7-a900-28027f31c7de/volumes/kubernetes.io~projected/kube-api-access-26d2p major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d40fd93-7004-40a5-a715-dcbac377dd94/volumes/kubernetes.io~projected/kube-api-access-grw5d:{mountpoint:/var/lib/kubelet/pods/4d40fd93-7004-40a5-a715-dcbac377dd94/volumes/kubernetes.io~projected/kube-api-access-grw5d major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5067ea99-b14b-4204-a4b4-9cfd0ae2f677/volumes/kubernetes.io~projected/kube-api-access-sl2wv:{mountpoint:/var/lib/kubelet/pods/5067ea99-b14b-4204-a4b4-9cfd0ae2f677/volumes/kubernetes.io~projected/kube-api-access-sl2wv major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60ef143c-9312-4bc0-8180-7864b729d2b2/volumes/kubernetes.io~projected/kube-api-access-68ttp:{mountpoint:/var/lib/kubelet/pods/60ef143c-9312-4bc0-8180-7864b729d2b2/volumes/kubernetes.io~projected/kube-api-access-68ttp major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/651ad3fc-4794-4a59-b41f-4fb01ed533d5/volumes/kubernetes.io~projected/kube-api-access-ffg8g:{mountpoint:/var/lib/kubelet/pods/651ad3fc-4794-4a59-b41f-4fb01ed533d5/volumes/kubernetes.io~projected/kube-api-access-ffg8g major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/651ad3fc-4794-4a59-b41f-4fb01ed533d5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/651ad3fc-4794-4a59-b41f-4fb01ed533d5/volumes/kubernetes.io~secret/cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66c2abab-7f39-4d34-8571-ad883dde9532/volumes/kubernetes.io~projected/kube-api-access-bsj84:{mountpoint:/var/lib/kubelet/pods/66c2abab-7f39-4d34-8571-ad883dde9532/volumes/kubernetes.io~projected/kube-api-access-bsj84 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66c2abab-7f39-4d34-8571-ad883dde9532/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/66c2abab-7f39-4d34-8571-ad883dde9532/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77135550-06c2-4f6b-8b7a-9ae0f81abb70/volumes/kubernetes.io~projected/kube-api-access-nzpbn:{mountpoint:/var/lib/kubelet/pods/77135550-06c2-4f6b-8b7a-9ae0f81abb70/volumes/kubernetes.io~projected/kube-api-access-nzpbn major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82d16c9b-8fb4-4d04-a896-ee08f8a616ce/volumes/kubernetes.io~projected/kube-api-access-bqs7c:{mountpoint:/var/lib/kubelet/pods/82d16c9b-8fb4-4d04-a896-ee08f8a616ce/volumes/kubernetes.io~projected/kube-api-access-bqs7c major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e6e6852-e747-4869-aea2-5e0bbc44988f/volumes/kubernetes.io~projected/kube-api-access-f742m:{mountpoint:/var/lib/kubelet/pods/8e6e6852-e747-4869-aea2-5e0bbc44988f/volumes/kubernetes.io~projected/kube-api-access-f742m major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f6d3f56-0c8f-4631-aed3-7dedc802377a/volumes/kubernetes.io~projected/kube-api-access-4hn6p:{mountpoint:/var/lib/kubelet/pods/8f6d3f56-0c8f-4631-aed3-7dedc802377a/volumes/kubernetes.io~projected/kube-api-access-4hn6p major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0698158-f320-40dc-8b65-bd56c57e4ac7/volumes/kubernetes.io~projected/kube-api-access-xsflx:{mountpoint:/var/lib/kubelet/pods/a0698158-f320-40dc-8b65-bd56c57e4ac7/volumes/kubernetes.io~projected/kube-api-access-xsflx major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a08862fb-0cb0-4e65-89b1-739c038bd82f/volumes/kubernetes.io~projected/kube-api-access-x2dxc:{mountpoint:/var/lib/kubelet/pods/a08862fb-0cb0-4e65-89b1-739c038bd82f/volumes/kubernetes.io~projected/kube-api-access-x2dxc major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3e70c22-8486-4834-8332-a8cf3e34d738/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a3e70c22-8486-4834-8332-a8cf3e34d738/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a88a6270-a93e-487d-a0b2-53a5b020c388/volumes/kubernetes.io~projected/kube-api-access-qhxb4:{mountpoint:/var/lib/kubelet/pods/a88a6270-a93e-487d-a0b2-53a5b020c388/volumes/kubernetes.io~projected/kube-api-access-qhxb4 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volumes/kubernetes.io~projected/kube-api-access-qcqc4:{mountpoint:/var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volumes/kubernetes.io~projected/kube-api-access-qcqc4 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc4a9361-8679-4e65-83f4-d8d876321372/volumes/kubernetes.io~projected/kube-api-access-2hvqx:{mountpoint:/var/lib/kubelet/pods/cc4a9361-8679-4e65-83f4-d8d876321372/volumes/kubernetes.io~projected/kube-api-access-2hvqx major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc4a9361-8679-4e65-83f4-d8d876321372/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/cc4a9361-8679-4e65-83f4-d8d876321372/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce59c590-a124-4655-9aaf-67e37b6e9fc6/volumes/kubernetes.io~projected/kube-api-access-64862:{mountpoint:/var/lib/kubelet/pods/ce59c590-a124-4655-9aaf-67e37b6e9fc6/volumes/kubernetes.io~projected/kube-api-access-64862 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd9fb448-f983-484e-85a4-afbd3efddd1d/volumes/kubernetes.io~projected/kube-api-access-c87mw:{mountpoint:/var/lib/kubelet/pods/dd9fb448-f983-484e-85a4-afbd3efddd1d/volumes/kubernetes.io~projected/kube-api-access-c87mw major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd9fb448-f983-484e-85a4-afbd3efddd1d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/dd9fb448-f983-484e-85a4-afbd3efddd1d/volumes/kubernetes.io~secret/serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e28966c5-8bda-4f1b-a66a-ba3493523a38/volumes/kubernetes.io~projected/kube-api-access-mz4gb:{mountpoint:/var/lib/kubelet/pods/e28966c5-8bda-4f1b-a66a-ba3493523a38/volumes/kubernetes.io~projected/kube-api-access-mz4gb major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e28966c5-8bda-4f1b-a66a-ba3493523a38/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/e28966c5-8bda-4f1b-a66a-ba3493523a38/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e66a1a72-e152-44d7-aa2b-846e8a872078/volumes/kubernetes.io~projected/kube-api-access-q77zm:{mountpoint:/var/lib/kubelet/pods/e66a1a72-e152-44d7-aa2b-846e8a872078/volumes/kubernetes.io~projected/kube-api-access-q77zm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1684862-0c18-4f2f-87fa-811ca427d4a5/volumes/kubernetes.io~projected/kube-api-access-mjvwn:{mountpoint:/var/lib/kubelet/pods/f1684862-0c18-4f2f-87fa-811ca427d4a5/volumes/kubernetes.io~projected/kube-api-access-mjvwn major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f264a363-68bf-4fc1-87c6-824a8796d7ef/volumes/kubernetes.io~projected/kube-api-access-45cgf:{mountpoint:/var/lib/kubelet/pods/f264a363-68bf-4fc1-87c6-824a8796d7ef/volumes/kubernetes.io~projected/kube-api-access-45cgf major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5cdff40-3fc5-423b-b14f-21172fab4f71/volumes/kubernetes.io~projected/kube-api-access-scq68:{mountpoint:/var/lib/kubelet/pods/f5cdff40-3fc5-423b-b14f-21172fab4f71/volumes/kubernetes.io~projected/kube-api-access-scq68 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fab1bb75-ccb3-4121-b885-4ab456c6ec6d/volumes/kubernetes.io~projected/kube-api-access-sgr68:{mountpoint:/var/lib/kubelet/pods/fab1bb75-ccb3-4121-b885-4ab456c6ec6d/volumes/kubernetes.io~projected/kube-api-access-sgr68 major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/db25638a09da0688a2967efa7a654cb42762ca74c7c0634e10b71e80cfee07f2/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fedd11e65755b30857fa01db61f86f39526ce908d4eeb0159a87f9b71862af0a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0e3092bf01e430da5f3a23dafe0f8e73d46735f08f6d8d70281b93a9d263bb3f/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/7060a65b489d4ef77785a7d44c405892a86048a73f2571b9f760d802c1cf2f12/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5534111d4ba5f4e453e5a7c07e1213bf29150802ce3c469e3168865e6acf3422/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c1ef0828edcf36c047385da4f64204fd59bbfc2fee0acd04f1a1242a6b0b1171/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/62fbb8fbac4fd1b64966c059d0ef39c8a255c0d28b950d5a7d821a7472a21927/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e571ef0b3e0c8597d445a1c173ca0a3a66ed0978e008cb38faaeadf280df5dd9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/cea8f36862f5ebb1485115956de98593dfd2bbce092747f6c651b7039b8f65ab/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5b0d85a98df863dd5854e323c25401b6e30ba65c35fb6f7c3abe156d7d7268af/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/96aa2c85749164328ef142da31c8b508467c4fb88d610e7fd7540deb2a3945d3/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/c9ca94bbee369d794df51bb4801e8f0b5164d479616af63740ad17ac97f391b6/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/bfa0186f9edfcac561ed98cec31721303d822109d23a47e1b896de47e7d0f385/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/1b4647a01f116d1302652446d58e2117f81e8f6c602abd915b6c0d9f828f1e8b/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2f1fa14e29e023b1d6382ef43d031a69c356e02e148e511b7b6658a0f641eb98/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/47bb6ff54c1d054f3386348fc12677dfaba9c6f22ef0793ec40f970aa8eef680/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d4a89f630ace00947755066dfc10fe5b8eec099f3c814de5788de70c25488ac3/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a5335b04148036c78e89c3f4f76eb9052f576aac297616d442ff06c4da08f2e8/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0870ebefc1ea4be3b7194c9efbc64d8b5116c97d94ab17561231d53478a3966c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5a8d6248744d709a41975bc99bb5c6fe39d6f7c508bbff3809ea58f029909165/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e95553d46348fe8775e9cdf960189aed1812bb783275f63b4fb3500979f092cb/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/522aeedb0c52a180992357df6c92f4cee3685f7f1551759d856e8ba34cf0446f/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c4201749683ad0e6f0e252c1fbd7a9c3f0693d8e81b351c1829fe046b5180aea/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/43de7a2998e1b3d3e7ca84c55e608c0560d5994d4b41f0e222e769dbac816ca5/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/abf271a480df30d33a62762f7b85f347935193cb16a3ad5baeec58ceeb8573fb/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/8b75ae28c347b1251b7756bbb5bc88334903f7be3734922cdc0a09c0750a998c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/371972692cb19ded1d7362e42722b1c738cd44ce10b82126290fdddcd27ebe4b/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/e94b0c27e59ef08e33779bbd82e428d1c5aa96013ba5f3ce326db5d55696b229/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/2a77ad1d0e0625ce735ed4b8c7f9d9e5d2832888fbec913fbf504f34c1ba3e6b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/3b208ee01049cc2bd4bbf08823da308ab1cea02bcc1bf0ed201aee8fdfe0c236/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/a6d287ca242ba999c48306f350d7b371d81d47e40f981e4ece5a915cbc0254fc/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7a266d568189731c2534848bbc48740d3dccb08570529547e426b65c32b3c7bf/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/bd3ff2a908532d990358810f6c1f73c93f4c21aa657f9c0fb899fac330cb8a85/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/2ae8bd5575051ca9d625c17c927f4042e432abec8f1ce27b676c74b5bd111c26/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/c1cdb4647a26f64535b52b7f6cce922b0449505c221744aa1704b5255e1bddf4/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/0e0447ae73315eafa2f7da6b1bfa233696210bee3776ad503afa6b1f4de997db/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/2efec3c1d185ebac47adae3f30188c141f1476b4125f8adc0a1400b905553c67/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0218b9e93ac67f7062712e028930870021bbe7d9356bf242795903eeaf1c2559/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/ac3bcfd182aebd859821e48528eaabf0642dff045ab501227ce03a54d1f24d7e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/d6477547e25891a4e40db5c7acb589b89f44336a90c40159bf3363aa2736d2a0/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/ff589f48bc4ed164d291921827a9bec536da783dc756cfa23672ebae627e584e/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/69b631dec0f5222868f52529aca1e091903664e9bd8eec1171a5377e39426d47/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/64e7952a7bc402ea2af2379f9243382209ddbd3a2fa3548b70a78588b2fab221/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/312df3f0d9a5ca1265ff40568a406a11865ae9a18eaa216f200fa0158a7cf4a4/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/d91db4ce7ab7733bbad34ca568cd998285a20461c6afacef48d71bbf276bceaf/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/8f5cb6e6626b39dd4c65bf82fdd52542080328be28150aef7c2393ae0a903c98/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/c98678eca1ba81e80db544747dd601c4bf329c68fbec6eefe6c37e5169201198/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/f774e90835119e1057d790b73b8ef108524c2d3c8f11f225418ce92d5d911a40/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/d23f3fd7a4ef1114f135f5d895c7e5d0bd2646b5223bcb10fbaa46c05a595b37/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/6ab0cb020c6392e2b62cd1da1c8d7e02a0e0660645ed834c5b4e9d5f690f7669/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/9b3fb9b871fd7b0b72cab34ebea1372b402d3197650446542ef65e1bee376750/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/f315378beee1a78b6f5759026d4a8efd434830c0aca3e4dc42bf4e1f9c87cee2/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/84fa58ae9655a99c96116f98d1e9e1bdc1bf0dc9106392f64f3fd8e0c4bd622a/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/6f66357455a7f1f062dcdbacd1decf1ab89df203a96055a5d53777eb7500c50c/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/3d5c5e82db4675e1ff9718d7b0740483cdd37182518ae56a2f34b2f087ab20da/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/7e9124c74c2ccc1c83c1da4e4b8d17c13145de44a41751439341ae5cc9c8efb4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/bcd809fc9e1e40da7456d7d9e6baffda4ee2c0e712514a33aa18984f2c9e58b6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/f3d4dc84c19c84798491500904e35ac8ab6f93f521bdfffbb94099fae91a9827/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0e5f4dfff726438b39f5937b675577bc0d11cf38ffb7c9c8f88737698026ee1e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/77921493b967c7df5089e79222fb2b8726306b224249ffa1a25dfb501860c99e/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/182ed7dcdce715b6ab8cb7f2ccab4b4871f511b51d119c965558485dbc61883d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/faf0ce82074932d8b92728488c242355577ffbd903b98cf7cba1fba928b3c740/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 14:57:25.897031 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.896709 5091 manager.go:217] Machine: {Timestamp:2026-11-02 14:57:25.89622702 +0000 UTC m=+0.060779551 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101721 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21bf8278ce86b47cfce9f28bf8964d SystemUUID:ec21bf82-78ce-86b4-7cfc-e9f28bf8964d BootID:7657d6de-108e-43fa-92ab-b95ee9a9348b Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3e70c22-8486-4834-8332-a8cf3e34d738/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volumes/kubernetes.io~projected/kube-api-access-qcqc4 DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0052fb2cfc79d40966ac2ce35931aaee17d2107bbde55c88d1e18cc80f76640f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13c7b469-cf39-4d61-8f78-6fe03ba07dde/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/474137bf42696cd5c4022adc56270af8af99a73916e37ae89e465b4d5cd9e3e5/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1a7f3fee5e3b490bbdd893569a6b1a9d2109c734357d5ec55ab3e19de6d2f3d/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:207 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb23dd447ebd2ab8c9ff3aacfe62b53f822feb0bfb37158c94a9e60710a0ca73/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd9fb448-f983-484e-85a4-afbd3efddd1d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f1684862-0c18-4f2f-87fa-811ca427d4a5/volumes/kubernetes.io~projected/kube-api-access-mjvwn DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4e3ebfec996ebe70960e6564b6b531170257a74a9cf3b83128ce2c32e36df98/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60ef143c-9312-4bc0-8180-7864b729d2b2/volumes/kubernetes.io~projected/kube-api-access-68ttp DeviceMajor:0 DeviceMinor:343 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66c2abab-7f39-4d34-8571-ad883dde9532/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c3a88eef549c38f94849f199070aa129db2317fb4f40f6fdf0a4208412bbe6a/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fe317c6-3f53-41ef-8448-1c453427bf9f/volumes/kubernetes.io~projected/kube-api-access-4r27x DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/651ad3fc-4794-4a59-b41f-4fb01ed533d5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0698158-f320-40dc-8b65-bd56c57e4ac7/volumes/kubernetes.io~projected/kube-api-access-xsflx DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce59c590-a124-4655-9aaf-67e37b6e9fc6/volumes/kubernetes.io~projected/kube-api-access-64862 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5067ea99-b14b-4204-a4b4-9cfd0ae2f677/volumes/kubernetes.io~projected/kube-api-access-sl2wv DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~projected/kube-api-access-mbvt6 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f6d3f56-0c8f-4631-aed3-7dedc802377a/volumes/kubernetes.io~projected/kube-api-access-4hn6p DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a2188699d0c49c799c80c6830665ed41f362768ed051dba21e138ed76863058/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57c06348ab5617c0e0de7cd57c8d819ec6c9e6b708eb1cc4e68e13ce09b78a04/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e28966c5-8bda-4f1b-a66a-ba3493523a38/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~projected/kube-api-access-lgbgt DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f264a363-68bf-4fc1-87c6-824a8796d7ef/volumes/kubernetes.io~projected/kube-api-access-45cgf DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/651ad3fc-4794-4a59-b41f-4fb01ed533d5/volumes/kubernetes.io~projected/kube-api-access-ffg8g DeviceMajor:0 DeviceMinor:227 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a88a6270-a93e-487d-a0b2-53a5b020c388/volumes/kubernetes.io~projected/kube-api-access-qhxb4 DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdb9d27a9ccea7d9d35f5378e1b6d5524deccb1601d24f6c38f7c6fc6977f96b/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15b996fa8e9ad9f339605eda7b9687267b834fe0dd740605ded6bf0e2cebf35e/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0470c1c6bd446b1fd74d9557fa0217cf112af3d967563057147bd6a3bd097561/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0991e750-1337-47d8-a151-1c2ed20797f6/volumes/kubernetes.io~projected/kube-api-access-5p8wb DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a08862fb-0cb0-4e65-89b1-739c038bd82f/volumes/kubernetes.io~projected/kube-api-access-x2dxc DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~projected/kube-api-access-6dc7f DeviceMajor:0 DeviceMinor:233 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba8da49a3a2625da2e9ac83ac68f32b8d5250ce7765581a0a5aa67e8e59d3527/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e28966c5-8bda-4f1b-a66a-ba3493523a38/volumes/kubernetes.io~projected/kube-api-access-mz4gb DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7448549e64b59ddac21d21955e797794ff9e9a5b6250bcfb437179a8bde9b661/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/108f7147-992f-44ce-8cc4-dd7f8cd2521d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:205 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9362b24e5973c3f6e1e0b070a06ad71290bb043324cb5cda7e17036bfa15fdc/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66c2abab-7f39-4d34-8571-ad883dde9532/volumes/kubernetes.io~projected/kube-api-access-bsj84 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e6e6852-e747-4869-aea2-5e0bbc44988f/volumes/kubernetes.io~projected/kube-api-access-f742m DeviceMajor:0 DeviceMinor:232 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68c3b6350e7d8ac7acff171b72b4bd5eaa5c4e37032a42b2497f83a7785f264e/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e66a1a72-e152-44d7-aa2b-846e8a872078/volumes/kubernetes.io~projected/kube-api-access-q77zm DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77135550-06c2-4f6b-8b7a-9ae0f81abb70/volumes/kubernetes.io~projected/kube-api-access-nzpbn DeviceMajor:0 DeviceMinor:228 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc4a9361-8679-4e65-83f4-d8d876321372/volumes/kubernetes.io~projected/kube-api-access-2hvqx DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~projected/kube-api-access-dsnm7 DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5cdff40-3fc5-423b-b14f-21172fab4f71/volumes/kubernetes.io~projected/kube-api-access-scq68 DeviceMajor:0 DeviceMinor:231 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29/volumes/kubernetes.io~projected/kube-api-access-ffq26 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/758c86bb4082868c6a3207d25025192e56e515770948c0e425ef053b935c75f2/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d40fd93-7004-40a5-a715-dcbac377dd94/volumes/kubernetes.io~projected/kube-api-access-grw5d DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/82d16c9b-8fb4-4d04-a896-ee08f8a616ce/volumes/kubernetes.io~projected/kube-api-access-bqs7c DeviceMajor:0 DeviceMinor:332 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cc4a9361-8679-4e65-83f4-d8d876321372/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff55518fa722bcdb37db1dba1f34b9e283b91cadf71f0e53966a734802c1f84b/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a497697a549d9ada166faefe4b1ed2fd5e601762d813469110f13b790f74ce3/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33f48086-60c0-4d87-94a0-4adf4788b971/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fab1bb75-ccb3-4121-b885-4ab456c6ec6d/volumes/kubernetes.io~projected/kube-api-access-sgr68 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c18129a61162abd60982fab866842e381b870f5d464583d2db0139f941f6050f/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c90af1af14495f14a7be37cc60ea73509de7f6b2b1308bdb8021d9c51b3600fc/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c2dae42d8d681968fa25789766b6e929f3c7a1babfeab8cdbef53b8f8611ce6d/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37e220680195ea0672cfe4a3fe2a2c077d60e716e80d6402fc2f01130f9dbab9/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa67ca68f0f6459995af4bdf29b9a408ff5776636780871f20aba23737f5dfe8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a16a23a-e4f9-4fd7-a900-28027f31c7de/volumes/kubernetes.io~projected/kube-api-access-26d2p DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/13c7b469-cf39-4d61-8f78-6fe03ba07dde/volumes/kubernetes.io~projected/kube-api-access-rbc4v DeviceMajor:0 DeviceMinor:223 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aedd60ea-b38d-4860-ace6-f5ecf7900581/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dd9fb448-f983-484e-85a4-afbd3efddd1d/volumes/kubernetes.io~projected/kube-api-access-c87mw DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0470c1c6bd446b1 MacAddress:86:4f:f6:b0:6e:fe Speed:10000 Mtu:8901} {Name:15b996fa8e9ad9f MacAddress:7e:4f:d3:6c:17:b9 Speed:10000 Mtu:8901} {Name:3c3a88eef549c38 MacAddress:ba:c1:ca:12:2d:ec Speed:10000 Mtu:8901} {Name:474137bf42696cd MacAddress:96:32:2e:d9:fc:a2 Speed:10000 Mtu:8901} {Name:57c06348ab5617c MacAddress:9e:b5:d9:d0:53:4a Speed:10000 Mtu:8901} {Name:68c3b6350e7d8ac MacAddress:ea:3d:0b:e2:ea:68 Speed:10000 Mtu:8901} {Name:6a2188699d0c49c MacAddress:8a:73:ee:36:09:03 Speed:10000 Mtu:8901} {Name:758c86bb4082868 MacAddress:46:b4:be:c4:b2:21 Speed:10000 Mtu:8901} {Name:7a497697a549d9a MacAddress:f6:6d:05:78:cf:63 Speed:10000 Mtu:8901} {Name:b4e3ebfec996ebe MacAddress:72:56:1b:a3:8f:17 Speed:10000 Mtu:8901} {Name:ba8da49a3a2625d MacAddress:16:98:61:5f:0b:f9 Speed:10000 Mtu:8901} {Name:bdb9d27a9ccea7d MacAddress:5a:a3:42:ce:e4:f2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e1:8d:7d:a7 Speed:0 Mtu:9001} {Name:br-int MacAddress:42:d9:a9:7d:00:ba Speed:0 Mtu:8901} {Name:c18129a61162abd MacAddress:86:96:df:19:91:6b Speed:10000 Mtu:8901} {Name:d9362b24e5973c3 MacAddress:ce:35:0d:74:78:dc Speed:10000 Mtu:8901} {Name:d955faf7c07f6e5 MacAddress:5e:67:48:79:51:1f Speed:10000 Mtu:8901} {Name:e1a7f3fee5e3b49 MacAddress:ea:7c:78:9f:5a:c2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e1:8d:7d:a7 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:9a:89:a3:fd:ce:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 14:57:25.897263 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897036 5091 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 14:57:25.897263 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897079 5091 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 14:57:25.897420 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897392 5091 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 14:57:25.897560 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897421 5091 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-229.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 14:57:25.897609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897562 5091 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 14:57:25.897609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897570 5091 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 14:57:25.897609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897581 5091 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:57:25.897609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897599 5091 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 14:57:25.897705 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897674 5091 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 14:57:25.897855 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897845 5091 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 14:57:25.897961 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897943 5091 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 14:57:25.898003 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897986 5091 kubelet.go:493] "Attempting to sync node with API server" Nov 02 14:57:25.898003 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.897996 5091 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 14:57:25.898067 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.898006 5091 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 14:57:25.898067 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.898015 5091 kubelet.go:405] "Adding apiserver pod source" Nov 02 14:57:25.898067 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.898024 5091 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 14:57:25.899111 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.899091 5091 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:57:25.899111 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.899110 5091 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:57:25.899644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.899622 5091 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 14:57:25.902100 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902083 5091 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 14:57:25.902100 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902097 5091 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 14:57:25.902208 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902197 5091 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 14:57:25.902445 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902431 5091 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 14:57:25.902510 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902494 5091 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 14:57:25.902653 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902641 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 14:57:25.902702 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902661 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 14:57:25.902702 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902686 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 14:57:25.902702 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902696 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902703 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902712 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902720 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902728 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902738 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902746 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902766 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 14:57:25.902799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902781 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 14:57:25.903083 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902827 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 14:57:25.903083 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.902837 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 14:57:25.906436 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.906419 5091 server.go:1263] "Started kubelet" Nov 02 14:57:25.906537 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.906456 5091 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 14:57:25.906629 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.906576 5091 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 14:57:25.906689 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.906655 5091 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 14:57:25.907052 ip-10-0-130-229 systemd[1]: Started Kubernetes Kubelet. Nov 02 14:57:25.907216 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.907097 5091 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 14:57:25.908836 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.908822 5091 server.go:317] "Adding debug handlers to kubelet server" Nov 02 14:57:25.913365 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913346 5091 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 14:57:25.913449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913363 5091 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 14:57:25.913449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913414 5091 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 14:51:45 +0000 UTC" deadline="2028-06-28 13:57:33.957083823 +0000 UTC" Nov 02 14:57:25.913449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913433 5091 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14495h0m8.043654315s" Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913464 5091 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913471 5091 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:25.913482 5091 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.913662 5091 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914365 5091 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914380 5091 factory.go:55] Registering systemd factory Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914393 5091 factory.go:223] Registration of the systemd container factory successfully Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914669 5091 factory.go:153] Registering CRI-O factory Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914685 5091 factory.go:223] Registration of the crio container factory successfully Nov 02 14:57:25.915449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914703 5091 factory.go:103] Registering Raw factory Nov 02 14:57:25.916260 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.914716 5091 manager.go:1199] Started watching for new ooms in manager Nov 02 14:57:25.916650 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.916633 5091 manager.go:319] Starting recovery of all containers Nov 02 14:57:25.917169 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917130 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60ef143c-9312-4bc0-8180-7864b729d2b2" volumeName="kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp" seLinuxMountContext="" Nov 02 14:57:25.917237 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917185 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="651ad3fc-4794-4a59-b41f-4fb01ed533d5" volumeName="kubernetes.io/projected/651ad3fc-4794-4a59-b41f-4fb01ed533d5-kube-api-access-ffg8g" seLinuxMountContext="" Nov 02 14:57:25.917237 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917201 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedd60ea-b38d-4860-ace6-f5ecf7900581" volumeName="kubernetes.io/projected/aedd60ea-b38d-4860-ace6-f5ecf7900581-kube-api-access-qcqc4" seLinuxMountContext="" Nov 02 14:57:25.917237 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917216 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9fb448-f983-484e-85a4-afbd3efddd1d" volumeName="kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-trusted-ca" seLinuxMountContext="" Nov 02 14:57:25.917237 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917230 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067ea99-b14b-4204-a4b4-9cfd0ae2f677" volumeName="kubernetes.io/projected/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-kube-api-access-sl2wv" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917244 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b68b41e-2bcf-4d2c-a881-dcd809b89fc5" volumeName="kubernetes.io/configmap/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-konnectivity-ca" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917257 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29" volumeName="kubernetes.io/configmap/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-iptables-alerter-script" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917285 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108f7147-992f-44ce-8cc4-dd7f8cd2521d" volumeName="kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-stats-auth" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917300 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108f7147-992f-44ce-8cc4-dd7f8cd2521d" volumeName="kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-default-certificate" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917345 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c7b469-cf39-4d61-8f78-6fe03ba07dde" volumeName="kubernetes.io/configmap/13c7b469-cf39-4d61-8f78-6fe03ba07dde-config" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917372 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c7b469-cf39-4d61-8f78-6fe03ba07dde" volumeName="kubernetes.io/projected/13c7b469-cf39-4d61-8f78-6fe03ba07dde-kube-api-access-rbc4v" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917387 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration" seLinuxMountContext="" Nov 02 14:57:25.917413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917401 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f48086-60c0-4d87-94a0-4adf4788b971" volumeName="kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-etc-tuned" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917424 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917439 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917453 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3e70c22-8486-4834-8332-a8cf3e34d738" volumeName="kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917475 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a88a6270-a93e-487d-a0b2-53a5b020c388" volumeName="kubernetes.io/projected/a88a6270-a93e-487d-a0b2-53a5b020c388-kube-api-access-qhxb4" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917548 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedd60ea-b38d-4860-ace6-f5ecf7900581" volumeName="kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-config" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917561 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1684862-0c18-4f2f-87fa-811ca427d4a5" volumeName="kubernetes.io/projected/f1684862-0c18-4f2f-87fa-811ca427d4a5-kube-api-access-mjvwn" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917583 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" volumeName="kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-binary-copy" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917597 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0991e750-1337-47d8-a151-1c2ed20797f6" volumeName="kubernetes.io/empty-dir/0991e750-1337-47d8-a151-1c2ed20797f6-tmp-dir" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917617 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29" volumeName="kubernetes.io/projected/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-kube-api-access-ffq26" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917632 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13c7b469-cf39-4d61-8f78-6fe03ba07dde" volumeName="kubernetes.io/secret/13c7b469-cf39-4d61-8f78-6fe03ba07dde-serving-cert" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917651 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917663 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d40fd93-7004-40a5-a715-dcbac377dd94" volumeName="kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917688 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedd60ea-b38d-4860-ace6-f5ecf7900581" volumeName="kubernetes.io/secret/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917696 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9fb448-f983-484e-85a4-afbd3efddd1d" volumeName="kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-config" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917703 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6d3f56-0c8f-4631-aed3-7dedc802377a" volumeName="kubernetes.io/projected/8f6d3f56-0c8f-4631-aed3-7dedc802377a-kube-api-access-4hn6p" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917730 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d16c9b-8fb4-4d04-a896-ee08f8a616ce" volumeName="kubernetes.io/empty-dir/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-data-volume" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917740 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b68b41e-2bcf-4d2c-a881-dcd809b89fc5" volumeName="kubernetes.io/secret/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-agent-certs" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917747 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5cdff40-3fc5-423b-b14f-21172fab4f71" volumeName="kubernetes.io/projected/f5cdff40-3fc5-423b-b14f-21172fab4f71-kube-api-access-scq68" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917762 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" volumeName="kubernetes.io/projected/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-kube-api-access-sgr68" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917770 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917785 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917802 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a16a23a-e4f9-4fd7-a900-28027f31c7de" volumeName="kubernetes.io/projected/4a16a23a-e4f9-4fd7-a900-28027f31c7de-kube-api-access-26d2p" seLinuxMountContext="" Nov 02 14:57:25.917818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917823 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6e6852-e747-4869-aea2-5e0bbc44988f" volumeName="kubernetes.io/configmap/8e6e6852-e747-4869-aea2-5e0bbc44988f-telemetry-config" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917831 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6d3f56-0c8f-4631-aed3-7dedc802377a" volumeName="kubernetes.io/empty-dir/8f6d3f56-0c8f-4631-aed3-7dedc802377a-tmp-dir" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917864 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108f7147-992f-44ce-8cc4-dd7f8cd2521d" volumeName="kubernetes.io/projected/108f7147-992f-44ce-8cc4-dd7f8cd2521d-kube-api-access-mbvt6" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917871 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917890 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c2abab-7f39-4d34-8571-ad883dde9532" volumeName="kubernetes.io/projected/66c2abab-7f39-4d34-8571-ad883dde9532-kube-api-access-bsj84" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917905 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9fb448-f983-484e-85a4-afbd3efddd1d" volumeName="kubernetes.io/projected/dd9fb448-f983-484e-85a4-afbd3efddd1d-kube-api-access-c87mw" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917935 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e66a1a72-e152-44d7-aa2b-846e8a872078" volumeName="kubernetes.io/projected/e66a1a72-e152-44d7-aa2b-846e8a872078-kube-api-access-q77zm" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917942 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60ef143c-9312-4bc0-8180-7864b729d2b2" volumeName="kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917954 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="651ad3fc-4794-4a59-b41f-4fb01ed533d5" volumeName="kubernetes.io/secret/651ad3fc-4794-4a59-b41f-4fb01ed533d5-cert" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917968 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77135550-06c2-4f6b-8b7a-9ae0f81abb70" volumeName="kubernetes.io/configmap/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-config" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917976 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e6e6852-e747-4869-aea2-5e0bbc44988f" volumeName="kubernetes.io/projected/8e6e6852-e747-4869-aea2-5e0bbc44988f-kube-api-access-f742m" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.917995 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce59c590-a124-4655-9aaf-67e37b6e9fc6" volumeName="kubernetes.io/projected/ce59c590-a124-4655-9aaf-67e37b6e9fc6-kube-api-access-64862" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918002 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f264a363-68bf-4fc1-87c6-824a8796d7ef" volumeName="kubernetes.io/projected/f264a363-68bf-4fc1-87c6-824a8796d7ef-kube-api-access-45cgf" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918009 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60ef143c-9312-4bc0-8180-7864b729d2b2" volumeName="kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918031 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918038 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a16a23a-e4f9-4fd7-a900-28027f31c7de" volumeName="kubernetes.io/configmap/4a16a23a-e4f9-4fd7-a900-28027f31c7de-serviceca" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918047 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e28966c5-8bda-4f1b-a66a-ba3493523a38" volumeName="kubernetes.io/secret/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-key" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918063 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f6d3f56-0c8f-4631-aed3-7dedc802377a" volumeName="kubernetes.io/configmap/8f6d3f56-0c8f-4631-aed3-7dedc802377a-config-volume" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918078 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918085 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2591c008-5186-479c-bbf6-a8a763d037cf" volumeName="kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918104 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918124 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918142 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067ea99-b14b-4204-a4b4-9cfd0ae2f677" volumeName="kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-daemon-config" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918158 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedd60ea-b38d-4860-ace6-f5ecf7900581" volumeName="kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-script-lib" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918165 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918172 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aedd60ea-b38d-4860-ace6-f5ecf7900581" volumeName="kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-env-overrides" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918186 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc4a9361-8679-4e65-83f4-d8d876321372" volumeName="kubernetes.io/configmap/cc4a9361-8679-4e65-83f4-d8d876321372-config" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918194 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fe317c6-3f53-41ef-8448-1c453427bf9f" volumeName="kubernetes.io/projected/0fe317c6-3f53-41ef-8448-1c453427bf9f-kube-api-access-4r27x" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918200 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f48086-60c0-4d87-94a0-4adf4788b971" volumeName="kubernetes.io/projected/33f48086-60c0-4d87-94a0-4adf4788b971-kube-api-access-lgbgt" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918214 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c2abab-7f39-4d34-8571-ad883dde9532" volumeName="kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-service-ca-bundle" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918243 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c2abab-7f39-4d34-8571-ad883dde9532" volumeName="kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-trusted-ca-bundle" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918250 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77135550-06c2-4f6b-8b7a-9ae0f81abb70" volumeName="kubernetes.io/projected/77135550-06c2-4f6b-8b7a-9ae0f81abb70-kube-api-access-nzpbn" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918264 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e28966c5-8bda-4f1b-a66a-ba3493523a38" volumeName="kubernetes.io/configmap/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-cabundle" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918271 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c2abab-7f39-4d34-8571-ad883dde9532" volumeName="kubernetes.io/secret/66c2abab-7f39-4d34-8571-ad883dde9532-serving-cert" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918284 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82d16c9b-8fb4-4d04-a896-ee08f8a616ce" volumeName="kubernetes.io/projected/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-api-access-bqs7c" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918311 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc4a9361-8679-4e65-83f4-d8d876321372" volumeName="kubernetes.io/secret/cc4a9361-8679-4e65-83f4-d8d876321372-serving-cert" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918325 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e28966c5-8bda-4f1b-a66a-ba3493523a38" volumeName="kubernetes.io/projected/e28966c5-8bda-4f1b-a66a-ba3493523a38-kube-api-access-mz4gb" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918346 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b735564-c5ef-4cc9-838c-6bd04cb9dc50" volumeName="kubernetes.io/configmap/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-nginx-conf" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918366 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60ef143c-9312-4bc0-8180-7864b729d2b2" volumeName="kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918373 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0698158-f320-40dc-8b65-bd56c57e4ac7" volumeName="kubernetes.io/projected/a0698158-f320-40dc-8b65-bd56c57e4ac7-kube-api-access-xsflx" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918379 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd9fb448-f983-484e-85a4-afbd3efddd1d" volumeName="kubernetes.io/secret/dd9fb448-f983-484e-85a4-afbd3efddd1d-serving-cert" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918395 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" volumeName="kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918402 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0991e750-1337-47d8-a151-1c2ed20797f6" volumeName="kubernetes.io/projected/0991e750-1337-47d8-a151-1c2ed20797f6-kube-api-access-5p8wb" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918409 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a8fa477-4b50-46af-b0cf-fe5801946921" volumeName="kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918422 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33f48086-60c0-4d87-94a0-4adf4788b971" volumeName="kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-tmp" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918447 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c2abab-7f39-4d34-8571-ad883dde9532" volumeName="kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-snapshots" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918461 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66c2abab-7f39-4d34-8571-ad883dde9532" volumeName="kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-tmp" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918475 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a08862fb-0cb0-4e65-89b1-739c038bd82f" volumeName="kubernetes.io/projected/a08862fb-0cb0-4e65-89b1-739c038bd82f-kube-api-access-x2dxc" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918482 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc4a9361-8679-4e65-83f4-d8d876321372" volumeName="kubernetes.io/projected/cc4a9361-8679-4e65-83f4-d8d876321372-kube-api-access-2hvqx" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918488 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" volumeName="kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918508 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5067ea99-b14b-4204-a4b4-9cfd0ae2f677" volumeName="kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cni-binary-copy" seLinuxMountContext="" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918535 5091 reconstruct.go:97] "Volume reconstruction finished" Nov 02 14:57:25.918861 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.918542 5091 reconciler.go:29] "Reconciler: start to sync state" Nov 02 14:57:25.938143 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.938117 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/1.log" Nov 02 14:57:25.938856 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.938776 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/1.log" Nov 02 14:57:25.941785 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.941747 5091 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 14:57:25.951919 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.951901 5091 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 14:57:25.951919 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.951919 5091 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 14:57:25.952062 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.951939 5091 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 14:57:25.952062 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:25.951994 5091 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 14:57:25.971001 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.970971 5091 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="3f6543a83842760e5ca2aab9f2bb5aba2e3620cf00dcdc5c7b7884282bf02ea4" exitCode=0 Nov 02 14:57:25.971001 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.971000 5091 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="5c4471d57b76be2c0b6f31a2cf1f8e99ecfda267b94e5dcfbdadd30071e1bbae" exitCode=0 Nov 02 14:57:25.971121 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.971013 5091 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="1d37ee1085995bcd8e59237cb6a3a0fd537e0c2437900e65d0a749889281880a" exitCode=0 Nov 02 14:57:25.971121 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.971023 5091 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="b6b19e3db97e5521de6c6f037405678a4f28d932bd6d4634459b7f31b50b95a1" exitCode=0 Nov 02 14:57:25.971121 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.971032 5091 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="9c726baec89f8bd72fd45bec31259775891fb236870a218eff778a280b916ec8" exitCode=0 Nov 02 14:57:25.971121 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.971041 5091 generic.go:356] "Generic (PLEG): container finished" podID="fab1bb75-ccb3-4121-b885-4ab456c6ec6d" containerID="3c7f3fbce41fd510fcd3ff08fea84281139490074ef2dc2786a23333158d7b34" exitCode=0 Nov 02 14:57:25.977355 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.977339 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/1.log" Nov 02 14:57:25.977450 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.977361 5091 generic.go:356] "Generic (PLEG): container finished" podID="dd9fb448-f983-484e-85a4-afbd3efddd1d" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" exitCode=255 Nov 02 14:57:25.983477 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:25.983455 5091 generic.go:356] "Generic (PLEG): container finished" podID="cb2ff2ec6d97130d91a627ec6e48c555" containerID="0b12ae965bd7ed6219b15e43661aa1a8a014b87a1f2f07b8c10da41957457141" exitCode=0 Nov 02 14:57:26.013580 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.013553 5091 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:57:26.019172 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.019157 5091 manager.go:324] Recovery completed Nov 02 14:57:26.036711 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.036660 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:26.040387 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.040364 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:26.040462 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.040403 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:26.040462 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.040415 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:26.043716 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043693 5091 cpu_manager.go:226] "Starting" policy="none" Nov 02 14:57:26.043716 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043713 5091 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 14:57:26.043871 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043734 5091 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 14:57:26.043926 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043906 5091 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 14:57:26.043969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043922 5091 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 14:57:26.043969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043951 5091 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 14:57:26.043969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043959 5091 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 14:57:26.043969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043967 5091 policy_none.go:50] "Start" Nov 02 14:57:26.044137 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043984 5091 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 14:57:26.044137 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.043997 5091 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 14:57:26.044251 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.044235 5091 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 14:57:26.044309 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.044252 5091 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 14:57:26.044309 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.044261 5091 policy_none.go:44] "Start" Nov 02 14:57:26.044763 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.044745 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:26.047455 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.047437 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:26.047541 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.047473 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:26.047541 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.047489 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:26.049530 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.049486 5091 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 14:57:26.052132 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052113 5091 manager.go:341] "Starting Device Plugin manager" Nov 02 14:57:26.052213 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.052145 5091 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 14:57:26.052213 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052157 5091 server.go:87] "Starting device plugin registration server" Nov 02 14:57:26.052213 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.052115 5091 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 14:57:26.052455 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052443 5091 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 14:57:26.052488 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052457 5091 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 14:57:26.052571 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052552 5091 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 14:57:26.052685 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052656 5091 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 14:57:26.052750 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052698 5091 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 14:57:26.052750 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052741 5091 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 14:57:26.052877 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.052795 5091 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T14:57:26.052684293Z","UUID":"3a71de3d-891c-4958-bc95-b9db046a2d55","Handler":null,"Name":"","Endpoint":""} Nov 02 14:57:26.054390 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.054372 5091 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 14:57:26.054575 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.054559 5091 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-229.us-west-2.compute.internal\" not found" Nov 02 14:57:26.054748 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.054725 5091 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 14:57:26.054871 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.054835 5091 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 14:57:26.103014 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.102990 5091 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:26.103150 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.103126 5091 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 14:57:26.103317 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.103302 5091 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:26.103648 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.103632 5091 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:26.151338 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.151313 5091 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 14:57:26.153259 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.153243 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 14:57:26.154475 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.154449 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 14:57:26.154608 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.154491 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 14:57:26.154608 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.154535 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-229.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 14:57:26.154608 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.154595 5091 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.252779 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.252731 5091 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal"] Nov 02 14:57:26.253353 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253321 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"548a2bc445bef7413f406b710579c42a","Type":"ContainerStarted","Data":"49a7681841195eaf95a5d8050844e0f4a98c7724ccd4db55e02a5627079eeab5"} Nov 02 14:57:26.253419 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253365 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"548a2bc445bef7413f406b710579c42a","Type":"ContainerStarted","Data":"0a61e299126f14a84360128de50cfa2801cd6603768fd5558ef649f75cb6dbc8"} Nov 02 14:57:26.253419 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253383 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"cb2ff2ec6d97130d91a627ec6e48c555","Type":"ContainerStarted","Data":"2dcce3ae79a83975bbc675f3150d09471cf45a9a87f54daee81dcc3d456cff51"} Nov 02 14:57:26.253419 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253400 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"cb2ff2ec6d97130d91a627ec6e48c555","Type":"ContainerDied","Data":"0b12ae965bd7ed6219b15e43661aa1a8a014b87a1f2f07b8c10da41957457141"} Nov 02 14:57:26.253539 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253419 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" event={"ID":"cb2ff2ec6d97130d91a627ec6e48c555","Type":"ContainerStarted","Data":"3a370bd314f79fa7c39a239ba8f0d62fbb49866555658eabb22d21b74259e35f"} Nov 02 14:57:26.253539 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253455 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.253607 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.253566 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.290949 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.290887 5091 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.290949 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.290946 5091 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.314868 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.314840 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:26.315001 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.314891 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.315480 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.315463 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:26.315552 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:26.315502 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.405225 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.405188 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548a2bc445bef7413f406b710579c42a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"548a2bc445bef7413f406b710579c42a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.405367 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.405245 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.405367 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.405276 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.506228 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.506194 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548a2bc445bef7413f406b710579c42a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"548a2bc445bef7413f406b710579c42a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.506446 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.506245 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.506446 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.506261 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.506446 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.506302 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.506446 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.506199 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/548a2bc445bef7413f406b710579c42a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"548a2bc445bef7413f406b710579c42a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.506446 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.506357 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cb2ff2ec6d97130d91a627ec6e48c555-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" (UID: \"cb2ff2ec6d97130d91a627ec6e48c555\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:26.898946 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:26.898903 5091 apiserver.go:52] "Watching apiserver" Nov 02 14:57:27.009361 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.009329 5091 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 14:57:27.010076 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.010044 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal","openshift-image-registry/node-ca-pq7wf","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf","kube-system/kas-connection-checker-685895c7c9-wfz5w","openshift-network-operator/iptables-alerter-7f792","kube-system/kas-connection-checker-685895c7c9-7s4b6","openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql","openshift-dns/dns-default-5xk7d","openshift-insights/periodic-gathering-2nf2p-5rnb4","kube-system/kas-connection-checker-685895c7c9-7hpmc","openshift-ingress-canary/ingress-canary-r4vdl","openshift-console-operator/console-operator-7cd56b8dbc-8vstz","openshift-dns/node-resolver-znsfq","kube-system/konnectivity-agent-48fls","openshift-network-console/networking-console-plugin-86c478bb88-nsrp7","openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z","openshift-multus/multus-additional-cni-plugins-wnkp5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc","openshift-image-registry/image-registry-59c9b7fcd6-bxrc7","openshift-network-diagnostics/network-check-target-hk7f2","openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq","openshift-service-ca/service-ca-5fffc97bdf-jpb5m","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f","openshift-ingress/router-default-54fb4b65f9-w4zp9","openshift-multus/network-metrics-daemon-56kkk","kube-system/global-pull-secret-syncer-88g4t","openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8","openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7","openshift-ovn-kubernetes/ovnkube-node-pw4p7","openshift-image-registry/image-registry-84798d88f5-xqqrs","openshift-multus/multus-xnz94","kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-kwbcz","openshift-insights/insights-operator-7c4986d449-r6gt7","openshift-insights/insights-runtime-extractor-r2l82"] Nov 02 14:57:27.011370 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.011350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.011745 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.011724 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.011996 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.011979 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.012398 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.012379 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.014987 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.014972 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:27.016678 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.016660 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:27.016759 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.016707 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:27.020958 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.020942 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:27.021977 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.021962 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.022375 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.022364 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.029027 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.029006 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.064228 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064205 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.064299 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064282 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 02 14:57:27.064427 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064409 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 02 14:57:27.064465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064438 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 02 14:57:27.064590 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064574 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 02 14:57:27.064644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064591 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 14:57:27.064689 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064679 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 02 14:57:27.064722 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064687 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.064831 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064818 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 14:57:27.064831 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064826 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 02 14:57:27.064917 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064827 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58zgw\"" Nov 02 14:57:27.064917 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064876 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.064917 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064906 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 14:57:27.065010 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.064930 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 02 14:57:27.066821 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.066805 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 14:57:27.069061 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069047 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.069138 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069084 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.069251 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069229 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 02 14:57:27.069251 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069238 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.069335 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069242 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.069335 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069264 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.069449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069381 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.069449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069411 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.069449 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069414 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.069575 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069462 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.069612 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069603 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.069687 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.069655 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.070232 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.070214 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 02 14:57:27.070232 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.070220 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 14:57:27.109183 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109157 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.109271 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109191 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbc4v\" (UniqueName: \"kubernetes.io/projected/13c7b469-cf39-4d61-8f78-6fe03ba07dde-kube-api-access-rbc4v\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:27.109271 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109210 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-default-certificate\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.109271 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-config\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.109386 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109334 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-tmp\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.109386 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.109450 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109388 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.109489 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109463 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-tmp\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.109568 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109485 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.109568 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-stats-auth\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.109729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109573 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvt6\" (UniqueName: \"kubernetes.io/projected/108f7147-992f-44ce-8cc4-dd7f8cd2521d-kube-api-access-mbvt6\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.109729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109600 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.109729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109627 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/77135550-06c2-4f6b-8b7a-9ae0f81abb70-kube-api-access-nzpbn\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.109729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13c7b469-cf39-4d61-8f78-6fe03ba07dde-serving-cert\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:27.109729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109677 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd9fb448-f983-484e-85a4-afbd3efddd1d-serving-cert\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109700 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66c2abab-7f39-4d34-8571-ad883dde9532-serving-cert\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109737 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-default-certificate\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109761 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45cgf\" (UniqueName: \"kubernetes.io/projected/f264a363-68bf-4fc1-87c6-824a8796d7ef-kube-api-access-45cgf\") pod \"kas-connection-checker-685895c7c9-7s4b6\" (UID: \"f264a363-68bf-4fc1-87c6-824a8796d7ef\") " pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e6e6852-e747-4869-aea2-5e0bbc44988f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109812 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f742m\" (UniqueName: \"kubernetes.io/projected/8e6e6852-e747-4869-aea2-5e0bbc44988f-kube-api-access-f742m\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109839 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvwn\" (UniqueName: \"kubernetes.io/projected/f1684862-0c18-4f2f-87fa-811ca427d4a5-kube-api-access-mjvwn\") pod \"volume-data-source-validator-58c9f5f46d-k262f\" (UID: \"f1684862-0c18-4f2f-87fa-811ca427d4a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-stats-auth\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109864 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc4a9361-8679-4e65-83f4-d8d876321372-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109911 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13c7b469-cf39-4d61-8f78-6fe03ba07dde-config\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109952 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.109992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.109981 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cc4a9361-8679-4e65-83f4-d8d876321372-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110052 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc4a9361-8679-4e65-83f4-d8d876321372-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110077 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsj84\" (UniqueName: \"kubernetes.io/projected/66c2abab-7f39-4d34-8571-ad883dde9532-kube-api-access-bsj84\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110100 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110117 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/8e6e6852-e747-4869-aea2-5e0bbc44988f-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110132 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110151 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-snapshots\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110190 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110221 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110240 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-service-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhxb4\" (UniqueName: \"kubernetes.io/projected/a88a6270-a93e-487d-a0b2-53a5b020c388-kube-api-access-qhxb4\") pod \"kas-connection-checker-685895c7c9-7hpmc\" (UID: \"a88a6270-a93e-487d-a0b2-53a5b020c388\") " pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110374 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110385 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/66c2abab-7f39-4d34-8571-ad883dde9532-snapshots\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110392 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110400 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110480 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110527 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hvqx\" (UniqueName: \"kubernetes.io/projected/cc4a9361-8679-4e65-83f4-d8d876321372-kube-api-access-2hvqx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87mw\" (UniqueName: \"kubernetes.io/projected/dd9fb448-f983-484e-85a4-afbd3efddd1d-kube-api-access-c87mw\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110586 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77zm\" (UniqueName: \"kubernetes.io/projected/e66a1a72-e152-44d7-aa2b-846e8a872078-kube-api-access-q77zm\") pod \"kas-connection-checker-685895c7c9-wfz5w\" (UID: \"e66a1a72-e152-44d7-aa2b-846e8a872078\") " pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110586 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110611 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.111329 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.110648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.115438 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.115418 5091 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 14:57:27.176342 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.176274 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.176644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.176628 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-gwmlb\"" Nov 02 14:57:27.176716 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.176668 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-pr6dp\"" Nov 02 14:57:27.177699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.177688 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 02 14:57:27.177799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.177786 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 02 14:57:27.179129 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.179112 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.179253 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.179238 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 14:57:27.179375 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.179360 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 14:57:27.179457 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.179404 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 02 14:57:27.179457 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.179422 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 02 14:57:27.179457 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.179426 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.180395 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180379 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cc4a9361-8679-4e65-83f4-d8d876321372-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:27.180490 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180476 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-service-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.180708 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180691 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w8kzp\"" Nov 02 14:57:27.180867 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180850 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r6rrw\"" Nov 02 14:57:27.180947 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180913 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cr8rg\"" Nov 02 14:57:27.181010 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180966 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4crbl\"" Nov 02 14:57:27.181010 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.180976 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-chwdw\"" Nov 02 14:57:27.181136 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.181121 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 14:57:27.181246 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.181236 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4d7td\"" Nov 02 14:57:27.181783 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.181769 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 14:57:27.181875 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.181829 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-xhs42\"" Nov 02 14:57:27.181932 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.181874 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kbp57\"" Nov 02 14:57:27.181932 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.181878 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-c2bv5\"" Nov 02 14:57:27.182141 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182129 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 14:57:27.182198 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182131 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dlx4j\"" Nov 02 14:57:27.182256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182243 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 02 14:57:27.182435 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182417 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 14:57:27.182530 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182501 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 14:57:27.182631 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182615 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 02 14:57:27.182680 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182627 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 02 14:57:27.182680 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182617 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182680 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182685 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-b9vbw\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182728 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tsn9p\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182742 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2tgnv\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182747 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d5546\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182762 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182756 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hnsn9\"" Nov 02 14:57:27.182777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182778 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182806 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182855 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q62fl\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182920 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gmvcm\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.182962 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s2tzl\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.183030 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-29n45\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.183117 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 14:57:27.183238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.183188 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ct4hh\"" Nov 02 14:57:27.184793 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.184776 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.184862 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.184818 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.184926 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.184907 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.184926 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.184923 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 14:57:27.185143 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.185128 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.185331 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.185315 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.185693 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.185678 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.190796 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.190772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dd9fb448-f983-484e-85a4-afbd3efddd1d-serving-cert\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.190883 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.190772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66c2abab-7f39-4d34-8571-ad883dde9532-serving-cert\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.190938 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.190906 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c2abab-7f39-4d34-8571-ad883dde9532-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.190938 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.190927 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13c7b469-cf39-4d61-8f78-6fe03ba07dde-serving-cert\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:27.211456 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211434 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-config\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.211507 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211462 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqc4\" (UniqueName: \"kubernetes.io/projected/aedd60ea-b38d-4860-ace6-f5ecf7900581-kube-api-access-qcqc4\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.211507 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-iptables-alerter-script\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.211581 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211503 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-slash\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.211581 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211560 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-bin\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.211660 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211642 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.211705 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211694 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cnibin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.211738 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211715 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.211768 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211753 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-script-lib\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.211821 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211802 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-registration-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.211856 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211846 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a16a23a-e4f9-4fd7-a900-28027f31c7de-serviceca\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.211890 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.211849 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:27.211890 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.211873 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:27.211890 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-system-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.211969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211893 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-netns\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.211969 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.211935 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.711916832 +0000 UTC m=+1.876469367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:27.211969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.211962 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-sys-fs\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.212057 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212012 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-env-overrides\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.212085 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212063 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-conf-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.212119 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212099 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-multus-certs\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.212223 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212209 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-etc-kubernetes\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.212256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212245 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scq68\" (UniqueName: \"kubernetes.io/projected/f5cdff40-3fc5-423b-b14f-21172fab4f71-kube-api-access-scq68\") pod \"network-check-source-6f78d59f99-p8v6z\" (UID: \"f5cdff40-3fc5-423b-b14f-21172fab4f71\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:27.212284 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212269 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.212313 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212300 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-hostroot\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.212313 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212302 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-env-overrides\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.212408 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212395 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-data-volume\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.212445 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.212476 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212465 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-data-volume\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.212542 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.212495 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:27.212542 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a16a23a-e4f9-4fd7-a900-28027f31c7de-host\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.212608 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212541 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4a16a23a-e4f9-4fd7-a900-28027f31c7de-serviceca\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.212608 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212564 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-multus\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.212665 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.212612 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.712599991 +0000 UTC m=+1.877152514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:27.212698 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212689 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-systemd-units\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.212729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212708 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-sys\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.212759 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212727 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.212759 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.212752 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-cabundle\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:27.213389 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6d3f56-0c8f-4631-aed3-7dedc802377a-tmp-dir\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.213389 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovn-node-metrics-cert\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.213389 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f6d3f56-0c8f-4631-aed3-7dedc802377a-tmp-dir\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.213389 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213325 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.213681 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213407 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-cabundle\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213737 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-binary-copy\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213788 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovn-node-metrics-cert\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.213946 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.213962 5091 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.214031 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.714014352 +0000 UTC m=+1.878566887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:27.214064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214031 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.214429 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214074 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.214429 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214162 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991e750-1337-47d8-a151-1c2ed20797f6-hosts-file\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.214429 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214196 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:27.214429 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214233 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.214429 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214357 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r27x\" (UniqueName: \"kubernetes.io/projected/0fe317c6-3f53-41ef-8448-1c453427bf9f-kube-api-access-4r27x\") pod \"migrator-66df5d496b-bk2l8\" (UID: \"0fe317c6-3f53-41ef-8448-1c453427bf9f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:27.214429 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214399 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.214707 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214439 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-key\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:27.214707 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214497 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg8g\" (UniqueName: \"kubernetes.io/projected/651ad3fc-4794-4a59-b41f-4fb01ed533d5-kube-api-access-ffg8g\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:27.214707 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.214707 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214607 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2wv\" (UniqueName: \"kubernetes.io/projected/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-kube-api-access-sl2wv\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.214707 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214668 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-kubernetes\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.214913 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214728 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsflx\" (UniqueName: \"kubernetes.io/projected/a0698158-f320-40dc-8b65-bd56c57e4ac7-kube-api-access-xsflx\") pod \"migrator-66df5d496b-zgzbq\" (UID: \"a0698158-f320-40dc-8b65-bd56c57e4ac7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:27.214913 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214771 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.214913 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214835 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8wb\" (UniqueName: \"kubernetes.io/projected/0991e750-1337-47d8-a151-1c2ed20797f6-kube-api-access-5p8wb\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.214913 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214870 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-etc-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.214913 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.214899 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.215160 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215041 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-log-socket\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.215160 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215129 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:27.215160 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215154 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.215304 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215190 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.215304 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215228 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.215304 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215261 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6d3f56-0c8f-4631-aed3-7dedc802377a-config-volume\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.215304 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215296 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.215466 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dxc\" (UniqueName: \"kubernetes.io/projected/a08862fb-0cb0-4e65-89b1-739c038bd82f-kube-api-access-x2dxc\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.215466 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215442 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-netd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.215576 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215503 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-conf\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.215576 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215556 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-crio-socket\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.215667 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215611 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs7c\" (UniqueName: \"kubernetes.io/projected/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-api-access-bqs7c\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.215667 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215648 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-agent-certs\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:27.215772 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215721 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-host-slash\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.215822 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215784 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.215867 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-modprobe-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.215984 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-run\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216039 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216100 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-daemon-config\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216146 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-netns\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-node-log\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216267 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216297 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-host\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216337 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216408 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26d2p\" (UniqueName: \"kubernetes.io/projected/4a16a23a-e4f9-4fd7-a900-28027f31c7de-kube-api-access-26d2p\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216442 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysconfig\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216495 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-systemd\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216580 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-kubelet\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216702 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64862\" (UniqueName: \"kubernetes.io/projected/ce59c590-a124-4655-9aaf-67e37b6e9fc6-kube-api-access-64862\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216782 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-kubelet-config\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.216853 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.217187 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e28966c5-8bda-4f1b-a66a-ba3493523a38-signing-key\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.217284 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.217331 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-agent-certs\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:27.218032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.217612 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218147 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218223 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hn6p\" (UniqueName: \"kubernetes.io/projected/8f6d3f56-0c8f-4631-aed3-7dedc802377a-kube-api-access-4hn6p\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-systemd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218334 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218366 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218399 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-var-lib-kubelet\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgbgt\" (UniqueName: \"kubernetes.io/projected/33f48086-60c0-4d87-94a0-4adf4788b971-kube-api-access-lgbgt\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218498 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cnibin\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-nginx-conf\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:27.218909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218869 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffq26\" (UniqueName: \"kubernetes.io/projected/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-kube-api-access-ffq26\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.219283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.218952 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.219283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219014 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-ovn\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.219283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.219283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219233 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-daemon-config\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.219283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-socket-dir-parent\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.219465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219306 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.219465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219326 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-k8s-cni-cncf-io\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.219465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219384 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-socket-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.219465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219408 5091 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 14:57:27.219465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219457 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-device-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.219651 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219576 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.219651 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.219619 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-etc-tuned\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.220106 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220081 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-konnectivity-ca\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:27.220193 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220120 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.220193 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220135 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-etc-tuned\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.220295 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220207 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4gb\" (UniqueName: \"kubernetes.io/projected/e28966c5-8bda-4f1b-a66a-ba3493523a38-kube-api-access-mz4gb\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:27.220295 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220266 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.220400 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991e750-1337-47d8-a151-1c2ed20797f6-tmp-dir\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.220400 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-tmp\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.220492 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220426 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0991e750-1337-47d8-a151-1c2ed20797f6-tmp-dir\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.220492 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220430 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33f48086-60c0-4d87-94a0-4adf4788b971-tmp\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.220492 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220455 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-dbus\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.220669 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.220492 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.720459397 +0000 UTC m=+1.885011937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:27.220669 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.220669 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220595 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.220819 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220712 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgr68\" (UniqueName: \"kubernetes.io/projected/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-kube-api-access-sgr68\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.220819 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220776 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:27.220896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220827 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/651ad3fc-4794-4a59-b41f-4fb01ed533d5-cert\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:27.220896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.220984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220926 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.220984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.220958 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.221065 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.221135 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cni-binary-copy\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.221186 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221173 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/651ad3fc-4794-4a59-b41f-4fb01ed533d5-cert\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:27.221235 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-kubelet\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.221294 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221236 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-var-lib-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.221294 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221258 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a3e70c22-8486-4834-8332-a8cf3e34d738-original-pull-secret\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.221361 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221318 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-bin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.221361 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221341 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cni-binary-copy\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.221452 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221365 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-lib-modules\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.221452 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.221406 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.222022 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.222001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/108f7147-992f-44ce-8cc4-dd7f8cd2521d-metrics-certs\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.225252 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.225233 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.226673 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.226654 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.226964 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.226949 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227769 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227819 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227824 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227819 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227909 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227926 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227911 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227928 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.227984 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.227959 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228082 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228080 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228104 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228132 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228167 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228196 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228214 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228199 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228239 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228281 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228293 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228300 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228342 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228347 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:27.228424 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228373 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 14:57:27.229122 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.228500 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:27.229122 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228573 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 14:57:27.229122 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228657 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-4xxxf\"" Nov 02 14:57:27.229122 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.228838 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 14:57:27.229122 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhxb4\" (UniqueName: \"kubernetes.io/projected/a88a6270-a93e-487d-a0b2-53a5b020c388-kube-api-access-qhxb4\") pod \"kas-connection-checker-685895c7c9-7hpmc\" (UID: \"a88a6270-a93e-487d-a0b2-53a5b020c388\") " pod="kube-system/kas-connection-checker-685895c7c9-7hpmc" Nov 02 14:57:27.229425 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229224 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45cgf\" (UniqueName: \"kubernetes.io/projected/f264a363-68bf-4fc1-87c6-824a8796d7ef-kube-api-access-45cgf\") pod \"kas-connection-checker-685895c7c9-7s4b6\" (UID: \"f264a363-68bf-4fc1-87c6-824a8796d7ef\") " pod="kube-system/kas-connection-checker-685895c7c9-7s4b6" Nov 02 14:57:27.229425 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229417 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbc4v\" (UniqueName: \"kubernetes.io/projected/13c7b469-cf39-4d61-8f78-6fe03ba07dde-kube-api-access-rbc4v\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:27.229633 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229611 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.229700 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229673 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77zm\" (UniqueName: \"kubernetes.io/projected/e66a1a72-e152-44d7-aa2b-846e8a872078-kube-api-access-q77zm\") pod \"kas-connection-checker-685895c7c9-wfz5w\" (UID: \"e66a1a72-e152-44d7-aa2b-846e8a872078\") " pod="kube-system/kas-connection-checker-685895c7c9-wfz5w" Nov 02 14:57:27.229751 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229693 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvwn\" (UniqueName: \"kubernetes.io/projected/f1684862-0c18-4f2f-87fa-811ca427d4a5-kube-api-access-mjvwn\") pod \"volume-data-source-validator-58c9f5f46d-k262f\" (UID: \"f1684862-0c18-4f2f-87fa-811ca427d4a5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-k262f" Nov 02 14:57:27.229824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229808 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-config\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.229888 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229858 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 14:57:27.229939 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.229884 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.229939 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.229890 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal" Nov 02 14:57:27.230040 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87mw\" (UniqueName: \"kubernetes.io/projected/dd9fb448-f983-484e-85a4-afbd3efddd1d-kube-api-access-c87mw\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.230166 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230145 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzpbn\" (UniqueName: \"kubernetes.io/projected/77135550-06c2-4f6b-8b7a-9ae0f81abb70-kube-api-access-nzpbn\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.230230 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hvqx\" (UniqueName: \"kubernetes.io/projected/cc4a9361-8679-4e65-83f4-d8d876321372-kube-api-access-2hvqx\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f\" (UID: \"cc4a9361-8679-4e65-83f4-d8d876321372\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" Nov 02 14:57:27.230230 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230213 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/13c7b469-cf39-4d61-8f78-6fe03ba07dde-config\") pod \"service-ca-operator-544d749c8c-67pql\" (UID: \"13c7b469-cf39-4d61-8f78-6fe03ba07dde\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" Nov 02 14:57:27.230330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230298 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvt6\" (UniqueName: \"kubernetes.io/projected/108f7147-992f-44ce-8cc4-dd7f8cd2521d-kube-api-access-mbvt6\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.230330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230327 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8b68b41e-2bcf-4d2c-a881-dcd809b89fc5-konnectivity-ca\") pod \"konnectivity-agent-48fls\" (UID: \"8b68b41e-2bcf-4d2c-a881-dcd809b89fc5\") " pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:27.230456 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230439 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f742m\" (UniqueName: \"kubernetes.io/projected/8e6e6852-e747-4869-aea2-5e0bbc44988f-kube-api-access-f742m\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.230563 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.230548 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/dd9fb448-f983-484e-85a4-afbd3efddd1d-trusted-ca\") pod \"console-operator-7cd56b8dbc-8vstz\" (UID: \"dd9fb448-f983-484e-85a4-afbd3efddd1d\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:27.232495 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.232475 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-config\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.232587 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.232573 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aedd60ea-b38d-4860-ace6-f5ecf7900581-ovnkube-script-lib\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.232674 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.232660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.232715 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.232680 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-iptables-alerter-script\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.234803 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.234786 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.237953 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.237935 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6d3f56-0c8f-4631-aed3-7dedc802377a-config-volume\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.239392 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.239376 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsj84\" (UniqueName: \"kubernetes.io/projected/66c2abab-7f39-4d34-8571-ad883dde9532-kube-api-access-bsj84\") pod \"insights-operator-7c4986d449-r6gt7\" (UID: \"66c2abab-7f39-4d34-8571-ad883dde9532\") " pod="openshift-insights/insights-operator-7c4986d449-r6gt7" Nov 02 14:57:27.269992 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.269971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqc4\" (UniqueName: \"kubernetes.io/projected/aedd60ea-b38d-4860-ace6-f5ecf7900581-kube-api-access-qcqc4\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.271828 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.271803 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scq68\" (UniqueName: \"kubernetes.io/projected/f5cdff40-3fc5-423b-b14f-21172fab4f71-kube-api-access-scq68\") pod \"network-check-source-6f78d59f99-p8v6z\" (UID: \"f5cdff40-3fc5-423b-b14f-21172fab4f71\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-p8v6z" Nov 02 14:57:27.280472 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.280449 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg8g\" (UniqueName: \"kubernetes.io/projected/651ad3fc-4794-4a59-b41f-4fb01ed533d5-kube-api-access-ffg8g\") pod \"ingress-canary-r4vdl\" (UID: \"651ad3fc-4794-4a59-b41f-4fb01ed533d5\") " pod="openshift-ingress-canary/ingress-canary-r4vdl" Nov 02 14:57:27.281448 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.281429 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dxc\" (UniqueName: \"kubernetes.io/projected/a08862fb-0cb0-4e65-89b1-739c038bd82f-kube-api-access-x2dxc\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.281448 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.281438 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2wv\" (UniqueName: \"kubernetes.io/projected/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-kube-api-access-sl2wv\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.281712 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.281691 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8wb\" (UniqueName: \"kubernetes.io/projected/0991e750-1337-47d8-a151-1c2ed20797f6-kube-api-access-5p8wb\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.281813 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.281799 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.282673 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.282657 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgbgt\" (UniqueName: \"kubernetes.io/projected/33f48086-60c0-4d87-94a0-4adf4788b971-kube-api-access-lgbgt\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.282924 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.282907 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r27x\" (UniqueName: \"kubernetes.io/projected/0fe317c6-3f53-41ef-8448-1c453427bf9f-kube-api-access-4r27x\") pod \"migrator-66df5d496b-bk2l8\" (UID: \"0fe317c6-3f53-41ef-8448-1c453427bf9f\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bk2l8" Nov 02 14:57:27.287032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.286704 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4gb\" (UniqueName: \"kubernetes.io/projected/e28966c5-8bda-4f1b-a66a-ba3493523a38-kube-api-access-mz4gb\") pod \"service-ca-5fffc97bdf-jpb5m\" (UID: \"e28966c5-8bda-4f1b-a66a-ba3493523a38\") " pod="openshift-service-ca/service-ca-5fffc97bdf-jpb5m" Nov 02 14:57:27.287032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.286716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grw5d\" (UniqueName: \"kubernetes.io/projected/4d40fd93-7004-40a5-a715-dcbac377dd94-kube-api-access-grw5d\") pod \"network-check-target-hk7f2\" (UID: \"4d40fd93-7004-40a5-a715-dcbac377dd94\") " pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:27.287032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.287000 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hn6p\" (UniqueName: \"kubernetes.io/projected/8f6d3f56-0c8f-4631-aed3-7dedc802377a-kube-api-access-4hn6p\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.287032 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.287016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgr68\" (UniqueName: \"kubernetes.io/projected/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-kube-api-access-sgr68\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.287202 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.287023 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffq26\" (UniqueName: \"kubernetes.io/projected/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-kube-api-access-ffq26\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.287202 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.287179 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") pod \"periodic-gathering-2nf2p-5rnb4\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:57:27.288064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.288047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs7c\" (UniqueName: \"kubernetes.io/projected/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-api-access-bqs7c\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.288334 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.288319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64862\" (UniqueName: \"kubernetes.io/projected/ce59c590-a124-4655-9aaf-67e37b6e9fc6-kube-api-access-64862\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:27.292507 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.292490 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.292578 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.292567 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsflx\" (UniqueName: \"kubernetes.io/projected/a0698158-f320-40dc-8b65-bd56c57e4ac7-kube-api-access-xsflx\") pod \"migrator-66df5d496b-zgzbq\" (UID: \"a0698158-f320-40dc-8b65-bd56c57e4ac7\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zgzbq" Nov 02 14:57:27.292947 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.292930 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26d2p\" (UniqueName: \"kubernetes.io/projected/4a16a23a-e4f9-4fd7-a900-28027f31c7de-kube-api-access-26d2p\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.315333 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.315315 5091 scope.go:122] "RemoveContainer" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" Nov 02 14:57:27.321835 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321816 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a16a23a-e4f9-4fd7-a900-28027f31c7de-host\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.321927 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321847 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-multus\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.321927 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321871 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-systemd-units\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.321927 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321893 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-sys\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.321927 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321916 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4a16a23a-e4f9-4fd7-a900-28027f31c7de-host\") pod \"node-ca-pq7wf\" (UID: \"4a16a23a-e4f9-4fd7-a900-28027f31c7de\") " pod="openshift-image-registry/node-ca-pq7wf" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-systemd-units\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-multus\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321973 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991e750-1337-47d8-a151-1c2ed20797f6-hosts-file\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.321992 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322000 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322018 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-sys\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0991e750-1337-47d8-a151-1c2ed20797f6-hosts-file\") pod \"node-resolver-znsfq\" (UID: \"0991e750-1337-47d8-a151-1c2ed20797f6\") " pod="openshift-dns/node-resolver-znsfq" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322042 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322067 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-kubernetes\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322085 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-etc-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322101 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.322109 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322111 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322119 5091 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322134 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-log-socket\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322146 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-kubernetes\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322153 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-etc-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322175 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.822158017 +0000 UTC m=+1.986710542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322189 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-log-socket\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322199 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.822184282 +0000 UTC m=+1.986736803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : configmap "kube-rbac-proxy" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322160 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322228 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322257 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322266 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.822255759 +0000 UTC m=+1.986808280 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322284 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-netd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322299 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-conf\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322301 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322307 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322315 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-crio-socket\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322323 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-netd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322335 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-host-slash\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322351 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322359 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-crio-socket\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322368 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-modprobe-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322384 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-run\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29-host-slash\") pod \"iptables-alerter-7f792\" (UID: \"0a4f0f98-450d-4bdf-aafc-f9eaaebf0f29\") " pod="openshift-network-operator/iptables-alerter-7f792" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322396 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.82238177 +0000 UTC m=+1.986934303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322405 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322429 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-conf\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-netns\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322454 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-run\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322464 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-modprobe-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322466 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-node-log\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-host\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322458 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-netns\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-node-log\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322555 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysconfig\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322575 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-host\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322582 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-systemd\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322598 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-kubelet\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.322623 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.822614454 +0000 UTC m=+1.987166971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : secret "metrics-daemon-secret" not found Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322626 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysconfig\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322642 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-kubelet\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322656 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-kubelet-config\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322665 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-systemd\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322683 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-systemd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.322682 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322705 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322744 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-systemd\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322750 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-var-lib-kubelet\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322751 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322778 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-run-ovn-kubernetes\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322778 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-kubelet-config\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322779 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cnibin\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322781 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-var-lib-kubelet\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322825 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-ovn\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322841 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-cnibin\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322847 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322864 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-run-ovn\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322871 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-socket-dir-parent\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322896 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-k8s-cni-cncf-io\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322867 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fab1bb75-ccb3-4121-b885-4ab456c6ec6d-system-cni-dir\") pod \"multus-additional-cni-plugins-wnkp5\" (UID: \"fab1bb75-ccb3-4121-b885-4ab456c6ec6d\") " pod="openshift-multus/multus-additional-cni-plugins-wnkp5" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322921 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-socket-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322920 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-socket-dir-parent\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322942 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-k8s-cni-cncf-io\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-device-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.322965 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-socket-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323006 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-dbus\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323024 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-device-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323036 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-kubelet\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-var-lib-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323060 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a3e70c22-8486-4834-8332-a8cf3e34d738-dbus\") pod \"global-pull-secret-syncer-88g4t\" (UID: \"a3e70c22-8486-4834-8332-a8cf3e34d738\") " pod="kube-system/global-pull-secret-syncer-88g4t" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323089 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-bin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323107 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-var-lib-openvswitch\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-lib-modules\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323111 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-kubelet\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323124 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-var-lib-cni-bin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323136 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-slash\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323134 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-etc-sysctl-d\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323154 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-bin\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323179 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-cni-bin\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323163 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aedd60ea-b38d-4860-ace6-f5ecf7900581-host-slash\") pod \"ovnkube-node-pw4p7\" (UID: \"aedd60ea-b38d-4860-ace6-f5ecf7900581\") " pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323192 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/33f48086-60c0-4d87-94a0-4adf4788b971-lib-modules\") pod \"tuned-kwbcz\" (UID: \"33f48086-60c0-4d87-94a0-4adf4788b971\") " pod="openshift-cluster-node-tuning-operator/tuned-kwbcz" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cnibin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323220 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323247 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-registration-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-cnibin\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323274 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-system-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-netns\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323313 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-sys-fs\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323323 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-system-cni-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323330 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-conf-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323299 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-registration-dir\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.323331 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-multus-certs\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-etc-kubernetes\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323391 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-netns\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323416 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-multus-conf-dir\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323439 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-host-run-multus-certs\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-hostroot\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-sys-fs\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323476 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-etc-kubernetes\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5067ea99-b14b-4204-a4b4-9cfd0ae2f677-hostroot\") pod \"multus-xnz94\" (UID: \"5067ea99-b14b-4204-a4b4-9cfd0ae2f677\") " pod="openshift-multus/multus-xnz94" Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.323504 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:27.823487974 +0000 UTC m=+1.988040493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:27.324256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.323536 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a08862fb-0cb0-4e65-89b1-739c038bd82f-etc-selinux\") pod \"aws-ebs-csi-driver-node-q5lxc\" (UID: \"a08862fb-0cb0-4e65-89b1-739c038bd82f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q5lxc" Nov 02 14:57:27.621232 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.621211 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pq7wf_4a16a23a-e4f9-4fd7-a900-28027f31c7de/node-ca/0.log" Nov 02 14:57:27.727267 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.727232 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:27.727407 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.727275 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:27.727407 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.727368 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:27.727407 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727383 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:27.727556 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.727407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:27.727556 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727434 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.727420569 +0000 UTC m=+2.891973109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:27.727556 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727438 5091 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:27.727556 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727485 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:27.727556 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727491 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.727472373 +0000 UTC m=+2.892024892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:27.727556 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727501 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:27.727739 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727566 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.727554954 +0000 UTC m=+2.892107486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:27.727739 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.727583 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.727577506 +0000 UTC m=+2.892130068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:27.828784 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.828752 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:27.828784 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.828784 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.828971 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.828808 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:27.828971 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.828891 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:27.828971 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.828915 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:27.828971 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.828942 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.828927099 +0000 UTC m=+2.993479634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.828983 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829000 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829035 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.829025207 +0000 UTC m=+2.993577724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829060 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.829060 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829111 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829132 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.829117018 +0000 UTC m=+2.993669568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:27.829184 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829152 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.829144541 +0000 UTC m=+2.993697060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : secret "metrics-daemon-secret" not found Nov 02 14:57:27.829417 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.829231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:27.829417 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829326 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:27.829417 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:27.829355 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:28.829347751 +0000 UTC m=+2.993900282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:27.829417 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:27.829380 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:28.025160 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.025090 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 14:57:28.025558 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.025490 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/1.log" Nov 02 14:57:28.025558 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.025531 5091 generic.go:356] "Generic (PLEG): container finished" podID="dd9fb448-f983-484e-85a4-afbd3efddd1d" containerID="dbbb9bb2f723704c7ecbc0ab5d87d9dd98ac1445fc177b2db6c84516f4fcb4f0" exitCode=255 Nov 02 14:57:28.025653 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.025600 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" event={"ID":"dd9fb448-f983-484e-85a4-afbd3efddd1d","Type":"ContainerDied","Data":"dbbb9bb2f723704c7ecbc0ab5d87d9dd98ac1445fc177b2db6c84516f4fcb4f0"} Nov 02 14:57:28.025702 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.025669 5091 scope.go:122] "RemoveContainer" containerID="fc2dc34e2a296ecdd233ac7cd81574bec258e3e11295d5ec09602af21d0c80a2" Nov 02 14:57:28.026080 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.026061 5091 scope.go:122] "RemoveContainer" containerID="dbbb9bb2f723704c7ecbc0ab5d87d9dd98ac1445fc177b2db6c84516f4fcb4f0" Nov 02 14:57:28.026217 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.026201 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8vstz_openshift-console-operator(dd9fb448-f983-484e-85a4-afbd3efddd1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" Nov 02 14:57:28.232277 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.232250 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-r4vdl_651ad3fc-4794-4a59-b41f-4fb01ed533d5/serve-healthcheck-canary/0.log" Nov 02 14:57:28.423151 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.423126 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-bk2l8_0fe317c6-3f53-41ef-8448-1c453427bf9f/migrator/0.log" Nov 02 14:57:28.622094 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.622063 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-zgzbq_a0698158-f320-40dc-8b65-bd56c57e4ac7/migrator/0.log" Nov 02 14:57:28.737445 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.737373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:28.737445 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.737412 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:28.737662 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.737461 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:28.737662 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.737488 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:28.737662 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737545 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.737506985 +0000 UTC m=+4.902059504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:28.737662 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737602 5091 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:28.737662 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737647 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:28.737830 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737649 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.737635936 +0000 UTC m=+4.902188468 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:28.737830 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737716 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.737694355 +0000 UTC m=+4.902246886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:28.737830 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737606 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:28.737830 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737735 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:28.737830 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.737795 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.737783284 +0000 UTC m=+4.902335805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:28.822309 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.822278 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f_cc4a9361-8679-4e65-83f4-d8d876321372/kube-storage-version-migrator-operator/0.log" Nov 02 14:57:28.838841 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.838813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:28.838926 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.838855 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:28.838926 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.838872 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:28.838926 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.838911 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:28.839051 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.838962 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:28.839051 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:28.838989 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:28.839051 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839018 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.839004991 +0000 UTC m=+5.003557532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:28.839051 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839027 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:28.839051 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839043 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839064 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839081 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.838969 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839098 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.839087622 +0000 UTC m=+5.003640154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839136 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.83912616 +0000 UTC m=+5.003678678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839148 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.839140285 +0000 UTC m=+5.003692802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : secret "metrics-daemon-secret" not found Nov 02 14:57:28.839214 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:28.839158 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert podName:2b735564-c5ef-4cc9-838c-6bd04cb9dc50 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:30.839153442 +0000 UTC m=+5.003705959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-nsrp7" (UID: "2b735564-c5ef-4cc9-838c-6bd04cb9dc50") : secret "networking-console-plugin-cert" not found Nov 02 14:57:29.029931 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:29.029858 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 14:57:29.465491 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:29.465463 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:30.759000 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.758962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:30.759000 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.759009 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.759128 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759137 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.759155 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759193 5091 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759222 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.759201639 +0000 UTC m=+8.923754161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759261 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759274 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84798d88f5-xqqrs: secret "image-registry-tls" not found Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759264 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls podName:77135550-06c2-4f6b-8b7a-9ae0f81abb70 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.759247654 +0000 UTC m=+8.923800173 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-z8spf" (UID: "77135550-06c2-4f6b-8b7a-9ae0f81abb70") : secret "samples-operator-tls" not found Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759357 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle podName:108f7147-992f-44ce-8cc4-dd7f8cd2521d nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.759331083 +0000 UTC m=+8.923883610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle") pod "router-default-54fb4b65f9-w4zp9" (UID: "108f7147-992f-44ce-8cc4-dd7f8cd2521d") : configmap references non-existent config key: service-ca.crt Nov 02 14:57:30.759483 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.759378 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls podName:2591c008-5186-479c-bbf6-a8a763d037cf nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.759367264 +0000 UTC m=+8.923919782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls") pod "image-registry-84798d88f5-xqqrs" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf") : secret "image-registry-tls" not found Nov 02 14:57:30.859746 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.859720 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:30.859887 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.859780 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:30.859887 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.859815 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:30.859887 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.859832 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:30.859887 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.859857 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.859863 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.859913 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.859898479 +0000 UTC m=+9.024450997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.859939 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.859948 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.859952 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59c9b7fcd6-bxrc7: secret "image-registry-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.859993 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls podName:8f6d3f56-0c8f-4631-aed3-7dedc802377a nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.859978598 +0000 UTC m=+9.024531130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls") pod "dns-default-5xk7d" (UID: "8f6d3f56-0c8f-4631-aed3-7dedc802377a") : secret "dns-default-metrics-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.860012 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls podName:2a8fa477-4b50-46af-b0cf-fe5801946921 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.860005122 +0000 UTC m=+9.024557640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls") pod "image-registry-59c9b7fcd6-bxrc7" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921") : secret "image-registry-tls" not found Nov 02 14:57:30.860038 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.860033 5091 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 02 14:57:30.860287 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:30.860062 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs podName:ce59c590-a124-4655-9aaf-67e37b6e9fc6 nodeName:}" failed. No retries permitted until 2026-11-02 14:57:34.860053066 +0000 UTC m=+9.024605588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs") pod "network-metrics-daemon-56kkk" (UID: "ce59c590-a124-4655-9aaf-67e37b6e9fc6") : secret "metrics-daemon-secret" not found Nov 02 14:57:30.862273 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.862252 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b735564-c5ef-4cc9-838c-6bd04cb9dc50-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-nsrp7\" (UID: \"2b735564-c5ef-4cc9-838c-6bd04cb9dc50\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:30.915327 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:30.915303 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" Nov 02 14:57:31.351845 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:31.351761 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-nsrp7"] Nov 02 14:57:31.358676 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:31.358645 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b735564_c5ef_4cc9_838c_6bd04cb9dc50.slice/crio-aae1b7af934f1d50fcb8e5bf3bb159a444e3ca28a80ce36f4d9bf15b66f12b80 WatchSource:0}: Error finding container aae1b7af934f1d50fcb8e5bf3bb159a444e3ca28a80ce36f4d9bf15b66f12b80: Status 404 returned error can't find the container with id aae1b7af934f1d50fcb8e5bf3bb159a444e3ca28a80ce36f4d9bf15b66f12b80 Nov 02 14:57:31.361388 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:31.361361 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 14:57:32.043283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:32.043202 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" event={"ID":"2b735564-c5ef-4cc9-838c-6bd04cb9dc50","Type":"ContainerStarted","Data":"aae1b7af934f1d50fcb8e5bf3bb159a444e3ca28a80ce36f4d9bf15b66f12b80"} Nov 02 14:57:32.047471 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:32.047444 5091 generic.go:356] "Generic (PLEG): container finished" podID="13c7b469-cf39-4d61-8f78-6fe03ba07dde" containerID="3b0de216106be53058fba5e77f2202ebcb1578d91a3b6456f314ba5a3da74245" exitCode=0 Nov 02 14:57:32.047595 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:32.047529 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" event={"ID":"13c7b469-cf39-4d61-8f78-6fe03ba07dde","Type":"ContainerDied","Data":"3b0de216106be53058fba5e77f2202ebcb1578d91a3b6456f314ba5a3da74245"} Nov 02 14:57:32.047863 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:32.047848 5091 scope.go:122] "RemoveContainer" containerID="3b0de216106be53058fba5e77f2202ebcb1578d91a3b6456f314ba5a3da74245" Nov 02 14:57:32.205085 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:32.205053 5091 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:32.205427 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:32.205413 5091 scope.go:122] "RemoveContainer" containerID="dbbb9bb2f723704c7ecbc0ab5d87d9dd98ac1445fc177b2db6c84516f4fcb4f0" Nov 02 14:57:32.205622 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:32.205603 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8vstz_openshift-console-operator(dd9fb448-f983-484e-85a4-afbd3efddd1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" Nov 02 14:57:33.053054 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:33.053021 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-67pql" event={"ID":"13c7b469-cf39-4d61-8f78-6fe03ba07dde","Type":"ContainerStarted","Data":"ca427d68aa339efd38932062f1f5aad10a9c623380a777ea38d82cbbc03f9e61"} Nov 02 14:57:33.564985 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:33.564955 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:34.057623 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.057590 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-nsrp7" event={"ID":"2b735564-c5ef-4cc9-838c-6bd04cb9dc50","Type":"ContainerStarted","Data":"4bd87e6f9288b20e277db8613ffa6b332a489622d3f5c693fe224b8e1e2bbf6b"} Nov 02 14:57:34.795547 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.795497 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:34.795732 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.795562 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:34.795732 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.795602 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:34.795732 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.795624 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:34.795895 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:34.795734 5091 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:34.795895 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:34.795800 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls podName:8e6e6852-e747-4869-aea2-5e0bbc44988f nodeName:}" failed. No retries permitted until 2026-11-02 14:57:42.795785608 +0000 UTC m=+16.960338129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-w8df7" (UID: "8e6e6852-e747-4869-aea2-5e0bbc44988f") : secret "cluster-monitoring-operator-tls" not found Nov 02 14:57:34.796179 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.796153 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/108f7147-992f-44ce-8cc4-dd7f8cd2521d-service-ca-bundle\") pod \"router-default-54fb4b65f9-w4zp9\" (UID: \"108f7147-992f-44ce-8cc4-dd7f8cd2521d\") " pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:34.798040 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.798018 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"image-registry-84798d88f5-xqqrs\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:34.798040 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.798036 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/77135550-06c2-4f6b-8b7a-9ae0f81abb70-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-z8spf\" (UID: \"77135550-06c2-4f6b-8b7a-9ae0f81abb70\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:34.811941 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.811920 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:34.812969 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.812954 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" Nov 02 14:57:34.813026 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.813017 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:34.898609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.897447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:34.898609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.897498 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:34.898609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.897543 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:34.898609 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.897572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:34.898999 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:34.898978 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 02 14:57:34.899177 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:34.899161 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls podName:82d16c9b-8fb4-4d04-a896-ee08f8a616ce nodeName:}" failed. No retries permitted until 2026-11-02 14:57:42.899136859 +0000 UTC m=+17.063689400 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls") pod "insights-runtime-extractor-r2l82" (UID: "82d16c9b-8fb4-4d04-a896-ee08f8a616ce") : secret "insights-runtime-extractor-tls" not found Nov 02 14:57:34.909189 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.909147 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"image-registry-59c9b7fcd6-bxrc7\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:34.911414 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.911361 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f6d3f56-0c8f-4631-aed3-7dedc802377a-metrics-tls\") pod \"dns-default-5xk7d\" (UID: \"8f6d3f56-0c8f-4631-aed3-7dedc802377a\") " pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:34.911853 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:34.911832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce59c590-a124-4655-9aaf-67e37b6e9fc6-metrics-certs\") pod \"network-metrics-daemon-56kkk\" (UID: \"ce59c590-a124-4655-9aaf-67e37b6e9fc6\") " pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:35.121907 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.121881 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-56kkk" Nov 02 14:57:35.122988 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.122970 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:35.123099 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.123079 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:35.197880 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.195162 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:35.197880 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.195735 5091 scope.go:122] "RemoveContainer" containerID="dbbb9bb2f723704c7ecbc0ab5d87d9dd98ac1445fc177b2db6c84516f4fcb4f0" Nov 02 14:57:35.197880 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:35.195974 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-8vstz_openshift-console-operator(dd9fb448-f983-484e-85a4-afbd3efddd1d)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" Nov 02 14:57:35.323777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.323752 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84798d88f5-xqqrs"] Nov 02 14:57:35.326906 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.326880 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf"] Nov 02 14:57:35.330488 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:35.330463 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2591c008_5186_479c_bbf6_a8a763d037cf.slice/crio-f69bdf437a10c2382b484ce00834c32e49353a40be9c43495349999bdaacd4c2 WatchSource:0}: Error finding container f69bdf437a10c2382b484ce00834c32e49353a40be9c43495349999bdaacd4c2: Status 404 returned error can't find the container with id f69bdf437a10c2382b484ce00834c32e49353a40be9c43495349999bdaacd4c2 Nov 02 14:57:35.333158 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.333137 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54fb4b65f9-w4zp9"] Nov 02 14:57:35.380476 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:35.380446 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108f7147_992f_44ce_8cc4_dd7f8cd2521d.slice/crio-4623f232ee3574af06ac86ff8b9b4b21c6c9ac6b852cda9ad5b46c8766c95322 WatchSource:0}: Error finding container 4623f232ee3574af06ac86ff8b9b4b21c6c9ac6b852cda9ad5b46c8766c95322: Status 404 returned error can't find the container with id 4623f232ee3574af06ac86ff8b9b4b21c6c9ac6b852cda9ad5b46c8766c95322 Nov 02 14:57:35.506874 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.506848 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-56kkk"] Nov 02 14:57:35.513867 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:35.513843 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce59c590_a124_4655_9aaf_67e37b6e9fc6.slice/crio-9556d5e1b2388ba10c423af374c004e3f93c9db8b2c39b94429bd383fe547e6a WatchSource:0}: Error finding container 9556d5e1b2388ba10c423af374c004e3f93c9db8b2c39b94429bd383fe547e6a: Status 404 returned error can't find the container with id 9556d5e1b2388ba10c423af374c004e3f93c9db8b2c39b94429bd383fe547e6a Nov 02 14:57:35.545278 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.545255 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59c9b7fcd6-bxrc7"] Nov 02 14:57:35.551631 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:35.551604 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8fa477_4b50_46af_b0cf_fe5801946921.slice/crio-3b95f79424dd4ce418531338c47dbe6cc66a530f7b9ed17396afe95d0aea3141 WatchSource:0}: Error finding container 3b95f79424dd4ce418531338c47dbe6cc66a530f7b9ed17396afe95d0aea3141: Status 404 returned error can't find the container with id 3b95f79424dd4ce418531338c47dbe6cc66a530f7b9ed17396afe95d0aea3141 Nov 02 14:57:35.556648 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:35.556624 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5xk7d"] Nov 02 14:57:35.563171 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:35.563149 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6d3f56_0c8f_4631_aed3_7dedc802377a.slice/crio-5bcd7ade9550d47aa981bb049e9438e10f0eddfde50e614968a5a476cebcb18a WatchSource:0}: Error finding container 5bcd7ade9550d47aa981bb049e9438e10f0eddfde50e614968a5a476cebcb18a: Status 404 returned error can't find the container with id 5bcd7ade9550d47aa981bb049e9438e10f0eddfde50e614968a5a476cebcb18a Nov 02 14:57:36.066197 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.066162 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" event={"ID":"2591c008-5186-479c-bbf6-a8a763d037cf","Type":"ContainerStarted","Data":"688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484"} Nov 02 14:57:36.066460 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.066212 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" event={"ID":"2591c008-5186-479c-bbf6-a8a763d037cf","Type":"ContainerStarted","Data":"f69bdf437a10c2382b484ce00834c32e49353a40be9c43495349999bdaacd4c2"} Nov 02 14:57:36.066460 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.066262 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:57:36.067376 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.067350 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xk7d" event={"ID":"8f6d3f56-0c8f-4631-aed3-7dedc802377a","Type":"ContainerStarted","Data":"5bcd7ade9550d47aa981bb049e9438e10f0eddfde50e614968a5a476cebcb18a"} Nov 02 14:57:36.069038 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.069014 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" event={"ID":"2a8fa477-4b50-46af-b0cf-fe5801946921","Type":"ContainerStarted","Data":"33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331"} Nov 02 14:57:36.069138 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.069054 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" event={"ID":"2a8fa477-4b50-46af-b0cf-fe5801946921","Type":"ContainerStarted","Data":"3b95f79424dd4ce418531338c47dbe6cc66a530f7b9ed17396afe95d0aea3141"} Nov 02 14:57:36.070072 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.070048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-56kkk" event={"ID":"ce59c590-a124-4655-9aaf-67e37b6e9fc6","Type":"ContainerStarted","Data":"9556d5e1b2388ba10c423af374c004e3f93c9db8b2c39b94429bd383fe547e6a"} Nov 02 14:57:36.071481 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.071453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" event={"ID":"108f7147-992f-44ce-8cc4-dd7f8cd2521d","Type":"ContainerStarted","Data":"f127ddd7833013171b8e13994d1dfe35cb3540b43d08a433e5cfea4784768183"} Nov 02 14:57:36.071587 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.071501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" event={"ID":"108f7147-992f-44ce-8cc4-dd7f8cd2521d","Type":"ContainerStarted","Data":"4623f232ee3574af06ac86ff8b9b4b21c6c9ac6b852cda9ad5b46c8766c95322"} Nov 02 14:57:36.072594 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.072577 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" event={"ID":"77135550-06c2-4f6b-8b7a-9ae0f81abb70","Type":"ContainerStarted","Data":"735e846f95396e2ed45ec49e12aa179baa75b5847f74011fc3b5c40ff346b726"} Nov 02 14:57:36.681625 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.680283 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:36.681625 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.680544 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:36.814154 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.814078 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:36.817157 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:36.816658 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:37.077661 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:37.077585 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:37.952988 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:37.952964 5091 scope.go:122] "RemoveContainer" containerID="dbbb9bb2f723704c7ecbc0ab5d87d9dd98ac1445fc177b2db6c84516f4fcb4f0" Nov 02 14:57:38.092706 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.092678 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 14:57:38.096920 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.096879 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" event={"ID":"dd9fb448-f983-484e-85a4-afbd3efddd1d","Type":"ContainerStarted","Data":"75bfe9f869f09c24afac362f44c03405d42a5c1ab865a5d3a6d2215071954a84"} Nov 02 14:57:38.097295 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.097274 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:38.098275 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.098253 5091 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-8vstz container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Nov 02 14:57:38.098335 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.098299 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" podUID="dd9fb448-f983-484e-85a4-afbd3efddd1d" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Nov 02 14:57:38.100487 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.100458 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-56kkk" event={"ID":"ce59c590-a124-4655-9aaf-67e37b6e9fc6","Type":"ContainerStarted","Data":"0c524a16b05810c5f663d8197ad0dd944ef8582513e9f5712e9c47166db9c734"} Nov 02 14:57:38.100624 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.100489 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:38.100624 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:38.100530 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-56kkk" event={"ID":"ce59c590-a124-4655-9aaf-67e37b6e9fc6","Type":"ContainerStarted","Data":"6363d18eeacb14729b24fffe0198000d918d94260b6f3010ab474ff130d2b00a"} Nov 02 14:57:39.109108 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:39.109076 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" event={"ID":"77135550-06c2-4f6b-8b7a-9ae0f81abb70","Type":"ContainerStarted","Data":"2aa0fb2d0023018cf88ff74832e2cc374225acab2fb79ee151c1f37562d5cb94"} Nov 02 14:57:39.110569 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:39.110541 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xk7d" event={"ID":"8f6d3f56-0c8f-4631-aed3-7dedc802377a","Type":"ContainerStarted","Data":"0021fbb919ee1696e6de7bcc71200d853fbae62988e2d9da88e3e46ad22bbaf7"} Nov 02 14:57:39.116241 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:39.115838 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-8vstz" Nov 02 14:57:40.118369 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:40.118331 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5xk7d" event={"ID":"8f6d3f56-0c8f-4631-aed3-7dedc802377a","Type":"ContainerStarted","Data":"8b02567c78b0ad554363b29c28d7a0f92f6ecb2f66230d4a8a9464359c33b175"} Nov 02 14:57:40.118848 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:40.118398 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:40.120274 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:40.120249 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-z8spf" event={"ID":"77135550-06c2-4f6b-8b7a-9ae0f81abb70","Type":"ContainerStarted","Data":"ccc44db551cb8feff83d726aa132a5366e7f36a76d5a082efe3a48901409e7fb"} Nov 02 14:57:42.747505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.747469 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:42.747945 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.747600 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 14:57:42.748874 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.748852 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54fb4b65f9-w4zp9" Nov 02 14:57:42.868061 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.868023 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:42.870381 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.870352 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/8e6e6852-e747-4869-aea2-5e0bbc44988f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-w8df7\" (UID: \"8e6e6852-e747-4869-aea2-5e0bbc44988f\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:42.912871 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.912847 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" Nov 02 14:57:42.969166 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.969138 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:42.971477 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:42.971454 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/82d16c9b-8fb4-4d04-a896-ee08f8a616ce-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r2l82\" (UID: \"82d16c9b-8fb4-4d04-a896-ee08f8a616ce\") " pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:43.229576 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:43.229547 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r2l82" Nov 02 14:57:43.311436 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:43.311406 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7"] Nov 02 14:57:43.318328 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:43.318299 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6e6852_e747_4869_aea2_5e0bbc44988f.slice/crio-1cd6c9810a8647fabfcce1413cee826f3ac99262049cce4ab47c27c4a4f9f359 WatchSource:0}: Error finding container 1cd6c9810a8647fabfcce1413cee826f3ac99262049cce4ab47c27c4a4f9f359: Status 404 returned error can't find the container with id 1cd6c9810a8647fabfcce1413cee826f3ac99262049cce4ab47c27c4a4f9f359 Nov 02 14:57:43.619499 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:43.619469 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r2l82"] Nov 02 14:57:43.626310 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:43.626285 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d16c9b_8fb4_4d04_a896_ee08f8a616ce.slice/crio-232b8646c7684f8aa01e5b79e018ca296d6ef0defd75b0d04e722ffb6777e26f WatchSource:0}: Error finding container 232b8646c7684f8aa01e5b79e018ca296d6ef0defd75b0d04e722ffb6777e26f: Status 404 returned error can't find the container with id 232b8646c7684f8aa01e5b79e018ca296d6ef0defd75b0d04e722ffb6777e26f Nov 02 14:57:43.751847 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:43.751822 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:57:44.140097 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.140058 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" event={"ID":"8e6e6852-e747-4869-aea2-5e0bbc44988f","Type":"ContainerStarted","Data":"1cd6c9810a8647fabfcce1413cee826f3ac99262049cce4ab47c27c4a4f9f359"} Nov 02 14:57:44.141496 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.141469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2l82" event={"ID":"82d16c9b-8fb4-4d04-a896-ee08f8a616ce","Type":"ContainerStarted","Data":"f535f6f4bce54f71ac0d40375e06e88315a41b9e81f5e7bd54db5eae6a792f60"} Nov 02 14:57:44.141635 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.141530 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2l82" event={"ID":"82d16c9b-8fb4-4d04-a896-ee08f8a616ce","Type":"ContainerStarted","Data":"232b8646c7684f8aa01e5b79e018ca296d6ef0defd75b0d04e722ffb6777e26f"} Nov 02 14:57:44.550328 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.550295 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-q4959"] Nov 02 14:57:44.576931 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.576900 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.597398 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.597373 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84798d88f5-xqqrs"] Nov 02 14:57:44.628695 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.628665 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 02 14:57:44.630207 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.630193 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 02 14:57:44.643782 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.643749 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zhtwk\"" Nov 02 14:57:44.646789 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.646768 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d7fd668ff-x8lhs"] Nov 02 14:57:44.658598 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.658577 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.677115 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.677096 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-q4959"] Nov 02 14:57:44.682844 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.682824 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a-tmp\") pod \"downloads-6dd789c8b7-q4959\" (UID: \"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a\") " pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.682933 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.682856 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxpck\" (UniqueName: \"kubernetes.io/projected/f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a-kube-api-access-qxpck\") pod \"downloads-6dd789c8b7-q4959\" (UID: \"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a\") " pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.700243 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.700222 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7fd668ff-x8lhs"] Nov 02 14:57:44.786993 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.786913 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxpck\" (UniqueName: \"kubernetes.io/projected/f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a-kube-api-access-qxpck\") pod \"downloads-6dd789c8b7-q4959\" (UID: \"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a\") " pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.787432 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787311 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9267ed0f-5060-468f-a6ed-41b0cd670452-trusted-ca\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787432 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787352 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9267ed0f-5060-468f-a6ed-41b0cd670452-installation-pull-secrets\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787565 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787438 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9267ed0f-5060-468f-a6ed-41b0cd670452-registry-certificates\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787565 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9267ed0f-5060-468f-a6ed-41b0cd670452-ca-trust-extracted\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787565 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-registry-tls\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787725 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9267ed0f-5060-468f-a6ed-41b0cd670452-image-registry-private-configuration\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787725 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzt8n\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-kube-api-access-kzt8n\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787725 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787668 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-bound-sa-token\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.787725 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.787699 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a-tmp\") pod \"downloads-6dd789c8b7-q4959\" (UID: \"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a\") " pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.788113 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.788093 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a-tmp\") pod \"downloads-6dd789c8b7-q4959\" (UID: \"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a\") " pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.850588 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.850555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxpck\" (UniqueName: \"kubernetes.io/projected/f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a-kube-api-access-qxpck\") pod \"downloads-6dd789c8b7-q4959\" (UID: \"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a\") " pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.888474 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888444 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9267ed0f-5060-468f-a6ed-41b0cd670452-registry-certificates\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9267ed0f-5060-468f-a6ed-41b0cd670452-ca-trust-extracted\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888511 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-registry-tls\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888559 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9267ed0f-5060-468f-a6ed-41b0cd670452-image-registry-private-configuration\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzt8n\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-kube-api-access-kzt8n\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888644 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888598 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-bound-sa-token\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888849 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888671 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9267ed0f-5060-468f-a6ed-41b0cd670452-trusted-ca\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888849 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888705 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9267ed0f-5060-468f-a6ed-41b0cd670452-installation-pull-secrets\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.888957 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.888898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9267ed0f-5060-468f-a6ed-41b0cd670452-ca-trust-extracted\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.889548 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.889488 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9267ed0f-5060-468f-a6ed-41b0cd670452-registry-certificates\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.890094 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.890068 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9267ed0f-5060-468f-a6ed-41b0cd670452-trusted-ca\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.891422 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.891398 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9267ed0f-5060-468f-a6ed-41b0cd670452-image-registry-private-configuration\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.891633 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.891617 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-registry-tls\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.891887 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.891871 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9267ed0f-5060-468f-a6ed-41b0cd670452-installation-pull-secrets\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.903888 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.903864 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:57:44.961069 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.960986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzt8n\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-kube-api-access-kzt8n\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.966896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.966868 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9267ed0f-5060-468f-a6ed-41b0cd670452-bound-sa-token\") pod \"image-registry-5d7fd668ff-x8lhs\" (UID: \"9267ed0f-5060-468f-a6ed-41b0cd670452\") " pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:44.977034 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:44.977006 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:45.373650 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:45.373622 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-q4959"] Nov 02 14:57:45.380284 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:45.380251 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c4b4c9_d7ff_486e_ac82_eaa0d4d4be5a.slice/crio-ab1a483fcde5fc3de0bfe890ea1cab4b33194297d29bdde816e9a446b4986130 WatchSource:0}: Error finding container ab1a483fcde5fc3de0bfe890ea1cab4b33194297d29bdde816e9a446b4986130: Status 404 returned error can't find the container with id ab1a483fcde5fc3de0bfe890ea1cab4b33194297d29bdde816e9a446b4986130 Nov 02 14:57:45.436361 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:45.436333 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d7fd668ff-x8lhs"] Nov 02 14:57:45.442886 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:45.442862 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9267ed0f_5060_468f_a6ed_41b0cd670452.slice/crio-9108eefb6e57a0f836315edfb56495bba5f68d8c0c0838f114361769d097518a WatchSource:0}: Error finding container 9108eefb6e57a0f836315edfb56495bba5f68d8c0c0838f114361769d097518a: Status 404 returned error can't find the container with id 9108eefb6e57a0f836315edfb56495bba5f68d8c0c0838f114361769d097518a Nov 02 14:57:46.156229 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.156193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2l82" event={"ID":"82d16c9b-8fb4-4d04-a896-ee08f8a616ce","Type":"ContainerStarted","Data":"af1b1900b194fc4f81c6dfa8e78f27c43f8f03d837e766203b339d59eb584445"} Nov 02 14:57:46.157415 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.157391 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-q4959" event={"ID":"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a","Type":"ContainerStarted","Data":"ab1a483fcde5fc3de0bfe890ea1cab4b33194297d29bdde816e9a446b4986130"} Nov 02 14:57:46.158898 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.158878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" event={"ID":"9267ed0f-5060-468f-a6ed-41b0cd670452","Type":"ContainerStarted","Data":"677232ac6895f7d716be662c57d1b61808900244429ffa81fb74cfacbbe9dda7"} Nov 02 14:57:46.159019 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.158910 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" event={"ID":"9267ed0f-5060-468f-a6ed-41b0cd670452","Type":"ContainerStarted","Data":"9108eefb6e57a0f836315edfb56495bba5f68d8c0c0838f114361769d097518a"} Nov 02 14:57:46.159019 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.159007 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:57:46.213627 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.213600 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-48fls" Nov 02 14:57:46.275740 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:46.275692 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" podStartSLOduration=2.275680789 podStartE2EDuration="2.275680789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:46.275424011 +0000 UTC m=+20.439976551" watchObservedRunningTime="2026-11-02 14:57:46.275680789 +0000 UTC m=+20.440233323" Nov 02 14:57:47.166935 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:47.166697 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-w8df7" event={"ID":"8e6e6852-e747-4869-aea2-5e0bbc44988f","Type":"ContainerStarted","Data":"f22da783b0e93e7987af0e198617aac436a663636082744934aaa582970317c4"} Nov 02 14:57:48.940849 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:48.940819 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw"] Nov 02 14:57:48.944430 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:48.944397 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:48.998899 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:48.998867 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw"] Nov 02 14:57:49.028858 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.028813 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6fc6839b-0cee-4266-ad3b-c9355dfd01e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2zfvw\" (UID: \"6fc6839b-0cee-4266-ad3b-c9355dfd01e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:49.041264 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.041233 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jx69b\"" Nov 02 14:57:49.041403 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.041296 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 02 14:57:49.129582 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.129551 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6fc6839b-0cee-4266-ad3b-c9355dfd01e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2zfvw\" (UID: \"6fc6839b-0cee-4266-ad3b-c9355dfd01e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:49.132390 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.132359 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6fc6839b-0cee-4266-ad3b-c9355dfd01e8-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2zfvw\" (UID: \"6fc6839b-0cee-4266-ad3b-c9355dfd01e8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:49.179536 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.179489 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r2l82" event={"ID":"82d16c9b-8fb4-4d04-a896-ee08f8a616ce","Type":"ContainerStarted","Data":"e4510d87886ea5ccf2733c06fa7631ebe05d9e76c2bbe44e81135c5a8bac7923"} Nov 02 14:57:49.256793 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.256757 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:49.635183 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:49.635137 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw"] Nov 02 14:57:49.642696 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:49.642665 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc6839b_0cee_4266_ad3b_c9355dfd01e8.slice/crio-8747da3adee6b1ad85be37ef6521ccbc833a9e367e3c14e30b9d676f79ff63b9 WatchSource:0}: Error finding container 8747da3adee6b1ad85be37ef6521ccbc833a9e367e3c14e30b9d676f79ff63b9: Status 404 returned error can't find the container with id 8747da3adee6b1ad85be37ef6521ccbc833a9e367e3c14e30b9d676f79ff63b9 Nov 02 14:57:50.127661 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.127632 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5xk7d" Nov 02 14:57:50.185396 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.185364 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" event={"ID":"6fc6839b-0cee-4266-ad3b-c9355dfd01e8","Type":"ContainerStarted","Data":"8747da3adee6b1ad85be37ef6521ccbc833a9e367e3c14e30b9d676f79ff63b9"} Nov 02 14:57:50.442031 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.441960 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-76868ccd4b-zltbl"] Nov 02 14:57:50.446492 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.446467 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.519661 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.519629 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 02 14:57:50.519824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.519744 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 02 14:57:50.519824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.519765 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tffmx\"" Nov 02 14:57:50.519824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.519793 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 02 14:57:50.519824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.519812 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 02 14:57:50.525094 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.525066 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76868ccd4b-zltbl"] Nov 02 14:57:50.544016 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.543985 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.544136 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.544023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.544136 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.544051 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.544224 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.544147 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.544266 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.544252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.544298 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.544281 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm9dv\" (UniqueName: \"kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.544343 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.544299 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.614491 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.614463 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.645574 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646418 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646443 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm9dv\" (UniqueName: \"kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646460 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.646508 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.647030 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.647197 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.647819 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.647319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.649260 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.649237 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.650140 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.650117 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.709873 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.709814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm9dv\" (UniqueName: \"kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv\") pod \"console-76868ccd4b-zltbl\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:50.760668 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:50.760629 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:57:51.146827 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:51.146793 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76868ccd4b-zltbl"] Nov 02 14:57:51.153488 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:57:51.153455 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3dc6a4_59c3_4d5b_af4c_5e94685902d1.slice/crio-0bc20bf581f66f782275517d00bf8ee4d4ce5c523921e31bb163603543c39c6d WatchSource:0}: Error finding container 0bc20bf581f66f782275517d00bf8ee4d4ce5c523921e31bb163603543c39c6d: Status 404 returned error can't find the container with id 0bc20bf581f66f782275517d00bf8ee4d4ce5c523921e31bb163603543c39c6d Nov 02 14:57:51.192894 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:51.192801 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76868ccd4b-zltbl" event={"ID":"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1","Type":"ContainerStarted","Data":"0bc20bf581f66f782275517d00bf8ee4d4ce5c523921e31bb163603543c39c6d"} Nov 02 14:57:52.200113 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:52.200072 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" event={"ID":"6fc6839b-0cee-4266-ad3b-c9355dfd01e8","Type":"ContainerStarted","Data":"9f5ae9be392d0d5ecd551c54d106f7f603e5a8a96228e0793651748ebc5d10f9"} Nov 02 14:57:52.200673 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:52.200401 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:52.201020 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:52.200996 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-pw4p7" Nov 02 14:57:52.206096 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:52.206076 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" Nov 02 14:57:52.313265 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:52.313206 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2zfvw" podStartSLOduration=2.680882689 podStartE2EDuration="4.313194284s" totalImagesPullingTime="1.632311595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:52.312283268 +0000 UTC m=+26.476835812" watchObservedRunningTime="2026-11-02 14:57:52.313194284 +0000 UTC m=+26.477746823" Nov 02 14:57:53.568261 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:53.568226 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hk7f2" Nov 02 14:57:53.692768 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:53.692738 5091 patch_prober.go:29] interesting pod/image-registry-59c9b7fcd6-bxrc7 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 14:57:53.692945 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:53.692784 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:57:53.756952 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:53.756918 5091 patch_prober.go:29] interesting pod/image-registry-59c9b7fcd6-bxrc7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 14:57:53.757101 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:53.756973 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:57:54.604067 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:54.604020 5091 patch_prober.go:29] interesting pod/image-registry-84798d88f5-xqqrs container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 02 14:57:54.604597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:54.604076 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" podUID="2591c008-5186-479c-bbf6-a8a763d037cf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 14:57:56.221287 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:56.221247 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76868ccd4b-zltbl" event={"ID":"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1","Type":"ContainerStarted","Data":"dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84"} Nov 02 14:57:56.344172 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:57:56.344118 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-76868ccd4b-zltbl" podStartSLOduration=2.189515151 podStartE2EDuration="6.344103104s" totalImagesPullingTime="4.154587953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:57:56.342381779 +0000 UTC m=+30.506934319" watchObservedRunningTime="2026-11-02 14:57:56.344103104 +0000 UTC m=+30.508655645" Nov 02 14:57:57.936933 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:57:57.936888 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 14:58:00.251563 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.251529 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zhvgg"] Nov 02 14:58:00.258877 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.258853 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.319010 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.318980 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 14:58:00.362648 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.362615 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zw4g4\"" Nov 02 14:58:00.362648 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.362619 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 14:58:00.368256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.368233 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 14:58:00.368256 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.368245 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 14:58:00.438033 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.437997 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438227 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438052 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-root\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438227 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438139 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-textfile\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438227 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-tls\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438391 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438269 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49c54e7f-83d6-4b42-a6e9-f4494a15206f-metrics-client-ca\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438391 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438354 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-sys\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438494 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438408 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sczq4\" (UniqueName: \"kubernetes.io/projected/49c54e7f-83d6-4b42-a6e9-f4494a15206f-kube-api-access-sczq4\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438494 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-wtmp\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.438494 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.438450 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539296 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539210 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539296 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539274 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-root\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539308 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-textfile\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539333 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-tls\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49c54e7f-83d6-4b42-a6e9-f4494a15206f-metrics-client-ca\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-sys\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539444 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sczq4\" (UniqueName: \"kubernetes.io/projected/49c54e7f-83d6-4b42-a6e9-f4494a15206f-kube-api-access-sczq4\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-wtmp\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539505 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539494 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.539944 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.539915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.540201 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.540177 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-textfile\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.540314 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.540244 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-root\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.540464 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.540433 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49c54e7f-83d6-4b42-a6e9-f4494a15206f-metrics-client-ca\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.540611 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.540590 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-sys\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.540742 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.540722 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-wtmp\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.542155 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.542134 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.542758 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.542738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49c54e7f-83d6-4b42-a6e9-f4494a15206f-node-exporter-tls\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.612016 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.611985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sczq4\" (UniqueName: \"kubernetes.io/projected/49c54e7f-83d6-4b42-a6e9-f4494a15206f-kube-api-access-sczq4\") pod \"node-exporter-zhvgg\" (UID: \"49c54e7f-83d6-4b42-a6e9-f4494a15206f\") " pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.761319 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.761280 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:58:00.761491 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.761358 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:58:00.767167 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.767137 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:58:00.880155 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:00.880123 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zhvgg" Nov 02 14:58:00.895297 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:58:00.895270 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c54e7f_83d6_4b42_a6e9_f4494a15206f.slice/crio-7e46432a63a47c1a63ab58a3112ce9011075133c9e7f22df67649aa683133ab5 WatchSource:0}: Error finding container 7e46432a63a47c1a63ab58a3112ce9011075133c9e7f22df67649aa683133ab5: Status 404 returned error can't find the container with id 7e46432a63a47c1a63ab58a3112ce9011075133c9e7f22df67649aa683133ab5 Nov 02 14:58:01.082079 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.082049 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5684f6796c-cxfkt"] Nov 02 14:58:01.087595 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.087569 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.192624 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.192551 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 02 14:58:01.194136 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.194110 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5684f6796c-cxfkt"] Nov 02 14:58:01.245490 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245447 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhvgg" event={"ID":"49c54e7f-83d6-4b42-a6e9-f4494a15206f","Type":"ContainerStarted","Data":"7e46432a63a47c1a63ab58a3112ce9011075133c9e7f22df67649aa683133ab5"} Nov 02 14:58:01.245705 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.245822 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245739 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.245822 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245781 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwvq\" (UniqueName: \"kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.245822 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.246042 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245836 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.246042 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245883 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.246042 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245907 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.246042 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.245933 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.250012 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.249992 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:58:01.347190 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347146 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347190 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347188 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347216 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347298 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347344 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347378 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwvq\" (UniqueName: \"kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347405 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.347729 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.348036 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347856 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.348036 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.347867 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.348249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.348231 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.348312 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.348295 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.348708 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.348446 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.350351 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.350332 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.350748 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.350731 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.413179 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.413145 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwvq\" (UniqueName: \"kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq\") pod \"console-5684f6796c-cxfkt\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:01.702641 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:01.702596 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:02.094896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:02.094817 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5684f6796c-cxfkt"] Nov 02 14:58:02.101609 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:58:02.101575 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07703925_bba5_4744_82af_6c35e9d7e5fd.slice/crio-9a3d17ec665165e0907b316219bb4e03e7153fb145bec9fd3992f1eae302e2d2 WatchSource:0}: Error finding container 9a3d17ec665165e0907b316219bb4e03e7153fb145bec9fd3992f1eae302e2d2: Status 404 returned error can't find the container with id 9a3d17ec665165e0907b316219bb4e03e7153fb145bec9fd3992f1eae302e2d2 Nov 02 14:58:02.253818 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:02.253781 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5684f6796c-cxfkt" event={"ID":"07703925-bba5-4744-82af-6c35e9d7e5fd","Type":"ContainerStarted","Data":"f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f"} Nov 02 14:58:02.253995 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:02.253837 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5684f6796c-cxfkt" event={"ID":"07703925-bba5-4744-82af-6c35e9d7e5fd","Type":"ContainerStarted","Data":"9a3d17ec665165e0907b316219bb4e03e7153fb145bec9fd3992f1eae302e2d2"} Nov 02 14:58:03.260777 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.260731 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-q4959" event={"ID":"f1c4b4c9-d7ff-486e-ac82-eaa0d4d4be5a","Type":"ContainerStarted","Data":"edbbfdaf3ea8bb2fa2add5224c712d0a1221fc32edb0bae618a8c04a544df40a"} Nov 02 14:58:03.261335 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.260846 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:58:03.263567 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.263534 5091 generic.go:356] "Generic (PLEG): container finished" podID="49c54e7f-83d6-4b42-a6e9-f4494a15206f" containerID="e01f8bd7d4cb4621746ff6b696fa5e748f8dbd19f195a48b59becbefd0b85da2" exitCode=0 Nov 02 14:58:03.263703 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.263622 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhvgg" event={"ID":"49c54e7f-83d6-4b42-a6e9-f4494a15206f","Type":"ContainerDied","Data":"e01f8bd7d4cb4621746ff6b696fa5e748f8dbd19f195a48b59becbefd0b85da2"} Nov 02 14:58:03.278016 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.277992 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-q4959" Nov 02 14:58:03.382694 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.382639 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-q4959" podStartSLOduration=2.204841458 podStartE2EDuration="19.38262091s" totalImagesPullingTime="17.177779452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:03.382303426 +0000 UTC m=+37.546855968" watchObservedRunningTime="2026-11-02 14:58:03.38262091 +0000 UTC m=+37.547173452" Nov 02 14:58:03.382981 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.382955 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5684f6796c-cxfkt" podStartSLOduration=2.382947165 podStartE2EDuration="2.382947165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:02.392695746 +0000 UTC m=+36.557248290" watchObservedRunningTime="2026-11-02 14:58:03.382947165 +0000 UTC m=+37.547499707" Nov 02 14:58:03.756699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:03.756664 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:58:04.272952 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:04.272900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhvgg" event={"ID":"49c54e7f-83d6-4b42-a6e9-f4494a15206f","Type":"ContainerStarted","Data":"f0dc403fe350025159065f2be2647a3088d698d2f465c01b9d63ee41fda219a4"} Nov 02 14:58:04.273432 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:04.272978 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhvgg" event={"ID":"49c54e7f-83d6-4b42-a6e9-f4494a15206f","Type":"ContainerStarted","Data":"37213d4d583d530aeb0e64387fc2409df625f2d3f9c5cedf2cfdec314f61179c"} Nov 02 14:58:04.392581 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:04.392509 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zhvgg" podStartSLOduration=2.586621562 podStartE2EDuration="4.392494705s" totalImagesPullingTime="1.72830924s" totalInitContainerRuntime="77.563903ms" isStatefulPod=true podCreationTimestamp="2026-11-02 14:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:04.390989625 +0000 UTC m=+38.555542367" watchObservedRunningTime="2026-11-02 14:58:04.392494705 +0000 UTC m=+38.557047246" Nov 02 14:58:04.604081 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:04.603998 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:58:07.171708 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:07.171618 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d7fd668ff-x8lhs" Nov 02 14:58:07.389687 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:07.389655 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59c9b7fcd6-bxrc7"] Nov 02 14:58:09.346005 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.345972 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5684f6796c-cxfkt"] Nov 02 14:58:09.393193 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.393162 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4cd949c9-85zr4"] Nov 02 14:58:09.433474 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.433448 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.462592 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.462546 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4cd949c9-85zr4"] Nov 02 14:58:09.527465 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527428 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527631 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527506 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527693 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527634 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527693 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527673 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527799 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527697 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527853 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527907 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527895 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.527983 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.527964 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcclz\" (UniqueName: \"kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.616912 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.616852 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" podUID="2591c008-5186-479c-bbf6-a8a763d037cf" containerName="registry" containerID="cri-o://688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484" gracePeriod=30 Nov 02 14:58:09.628920 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.628894 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.629041 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.628947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcclz\" (UniqueName: \"kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.629041 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.628980 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.629163 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.629058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630008 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.629979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630126 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630060 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630126 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630117 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630241 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630166 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630241 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630189 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630241 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630210 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630571 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.630931 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.631026 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.630955 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.631701 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.631681 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.632353 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.632333 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.694407 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.694381 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcclz\" (UniqueName: \"kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz\") pod \"console-6c4cd949c9-85zr4\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.775361 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.775155 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:09.941796 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:09.941764 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:58:10.037934 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.037899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038118 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.037977 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038118 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038011 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038118 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038044 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038118 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038088 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038343 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038129 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038343 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038194 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038343 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") pod \"2591c008-5186-479c-bbf6-a8a763d037cf\" (UID: \"2591c008-5186-479c-bbf6-a8a763d037cf\") " Nov 02 14:58:10.038831 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.038667 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:10.039709 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.039681 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:10.041069 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.041020 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "kube-api-access-6dc7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:10.041551 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.041481 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:10.041794 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.041768 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:10.042423 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.042397 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:10.042573 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.042541 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:10.047201 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.047179 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted" pod "2591c008-5186-479c-bbf6-a8a763d037cf" (UID: "2591c008-5186-479c-bbf6-a8a763d037cf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:10.139753 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.139673 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dc7f\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-kube-api-access-6dc7f\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.139973 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.139949 5091 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-installation-pull-secrets\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.140074 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.139976 5091 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-registry-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.140074 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.139992 5091 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-registry-certificates\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.140074 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.140006 5091 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2591c008-5186-479c-bbf6-a8a763d037cf-image-registry-private-configuration\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.140074 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.140020 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2591c008-5186-479c-bbf6-a8a763d037cf-trusted-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.140074 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.140035 5091 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2591c008-5186-479c-bbf6-a8a763d037cf-ca-trust-extracted\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.140074 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.140049 5091 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2591c008-5186-479c-bbf6-a8a763d037cf-bound-sa-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:10.234238 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.234200 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4cd949c9-85zr4"] Nov 02 14:58:10.241006 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:58:10.240975 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda084235b_a3c7_4215_ae03_cd225c30f1b9.slice/crio-076768219f44b7d1627943333e7cc07e18e96746c92b0fcfb1c2dc20c180481d WatchSource:0}: Error finding container 076768219f44b7d1627943333e7cc07e18e96746c92b0fcfb1c2dc20c180481d: Status 404 returned error can't find the container with id 076768219f44b7d1627943333e7cc07e18e96746c92b0fcfb1c2dc20c180481d Nov 02 14:58:10.307594 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.307562 5091 generic.go:356] "Generic (PLEG): container finished" podID="2591c008-5186-479c-bbf6-a8a763d037cf" containerID="688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484" exitCode=0 Nov 02 14:58:10.307738 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.307630 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" Nov 02 14:58:10.307738 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.307643 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" event={"ID":"2591c008-5186-479c-bbf6-a8a763d037cf","Type":"ContainerDied","Data":"688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484"} Nov 02 14:58:10.307738 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.307708 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84798d88f5-xqqrs" event={"ID":"2591c008-5186-479c-bbf6-a8a763d037cf","Type":"ContainerDied","Data":"f69bdf437a10c2382b484ce00834c32e49353a40be9c43495349999bdaacd4c2"} Nov 02 14:58:10.307879 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.307738 5091 scope.go:122] "RemoveContainer" containerID="688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484" Nov 02 14:58:10.309760 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.309729 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cd949c9-85zr4" event={"ID":"a084235b-a3c7-4215-ae03-cd225c30f1b9","Type":"ContainerStarted","Data":"076768219f44b7d1627943333e7cc07e18e96746c92b0fcfb1c2dc20c180481d"} Nov 02 14:58:10.319394 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.319372 5091 scope.go:122] "RemoveContainer" containerID="688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484" Nov 02 14:58:10.319726 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:10.319705 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484\": container with ID starting with 688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484 not found: ID does not exist" containerID="688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484" Nov 02 14:58:10.319814 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.319734 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484"} err="failed to get container status \"688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484\": rpc error: code = NotFound desc = could not find container \"688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484\": container with ID starting with 688c5cc5614fe8b586b5720ce4bf9dbe22a7d613e71768221fd48458dacf9484 not found: ID does not exist" Nov 02 14:58:10.477923 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.477848 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84798d88f5-xqqrs"] Nov 02 14:58:10.485364 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:10.485337 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84798d88f5-xqqrs"] Nov 02 14:58:11.316603 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:11.316569 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cd949c9-85zr4" event={"ID":"a084235b-a3c7-4215-ae03-cd225c30f1b9","Type":"ContainerStarted","Data":"9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240"} Nov 02 14:58:11.434323 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:11.434277 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c4cd949c9-85zr4" podStartSLOduration=2.434262778 podStartE2EDuration="2.434262778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:11.433541763 +0000 UTC m=+45.598094304" watchObservedRunningTime="2026-11-02 14:58:11.434262778 +0000 UTC m=+45.598815318" Nov 02 14:58:11.702786 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:11.702751 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:11.958842 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:11.958756 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2591c008-5186-479c-bbf6-a8a763d037cf" path="/var/lib/kubelet/pods/2591c008-5186-479c-bbf6-a8a763d037cf/volumes" Nov 02 14:58:13.778138 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.778107 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4cd949c9-85zr4"] Nov 02 14:58:13.825200 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.825173 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f7bbb9-6xxvm"] Nov 02 14:58:13.825548 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.825534 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2591c008-5186-479c-bbf6-a8a763d037cf" containerName="registry" Nov 02 14:58:13.825598 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.825549 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2591c008-5186-479c-bbf6-a8a763d037cf" containerName="registry" Nov 02 14:58:13.825635 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.825616 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2591c008-5186-479c-bbf6-a8a763d037cf" containerName="registry" Nov 02 14:58:13.897923 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.897898 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f7bbb9-6xxvm"] Nov 02 14:58:13.898064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.897996 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.973806 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.973774 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.973949 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.973813 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.973949 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.973830 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.973949 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.973938 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr5gp\" (UniqueName: \"kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.974079 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.973959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.974079 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.973986 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.974079 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.974001 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:13.974079 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:13.974019 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.074945 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.074877 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr5gp\" (UniqueName: \"kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.074945 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.074912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075125 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.074962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075125 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.074993 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075125 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.075017 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075291 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.075265 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075349 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.075326 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075403 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.075353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075463 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.075431 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.075896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.075871 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.076157 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.076134 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.076249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.076134 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.076249 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.076206 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.077616 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.077597 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.077709 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.077682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.143281 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.143248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr5gp\" (UniqueName: \"kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp\") pod \"console-75f7bbb9-6xxvm\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.225748 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.225717 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:14.633150 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:14.633109 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f7bbb9-6xxvm"] Nov 02 14:58:14.639181 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:58:14.639152 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb3a1ce_8c03_4418_8d09_390269d024d7.slice/crio-ff57ab1adeb74e50240dca66b85c32fb26fc70df01e355a12628512d655d94e0 WatchSource:0}: Error finding container ff57ab1adeb74e50240dca66b85c32fb26fc70df01e355a12628512d655d94e0: Status 404 returned error can't find the container with id ff57ab1adeb74e50240dca66b85c32fb26fc70df01e355a12628512d655d94e0 Nov 02 14:58:15.338980 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:15.338946 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f7bbb9-6xxvm" event={"ID":"2fb3a1ce-8c03-4418-8d09-390269d024d7","Type":"ContainerStarted","Data":"407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e"} Nov 02 14:58:15.339443 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:15.338995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f7bbb9-6xxvm" event={"ID":"2fb3a1ce-8c03-4418-8d09-390269d024d7","Type":"ContainerStarted","Data":"ff57ab1adeb74e50240dca66b85c32fb26fc70df01e355a12628512d655d94e0"} Nov 02 14:58:15.452327 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:15.452278 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75f7bbb9-6xxvm" podStartSLOduration=2.452264682 podStartE2EDuration="2.452264682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:15.452001314 +0000 UTC m=+49.616553854" watchObservedRunningTime="2026-11-02 14:58:15.452264682 +0000 UTC m=+49.616817221" Nov 02 14:58:19.775326 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:19.775292 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:24.226737 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:24.226694 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:24.227199 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:24.226767 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:24.231457 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:24.231435 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:24.379812 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:24.379785 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:58:24.525846 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:24.525772 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76868ccd4b-zltbl"] Nov 02 14:58:27.933355 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:27.933313 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 14:58:32.413345 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.413282 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerName="registry" containerID="cri-o://33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331" gracePeriod=30 Nov 02 14:58:32.722824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.722798 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:58:32.845478 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845452 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.845478 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845486 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.845683 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845540 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.845683 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.845764 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845709 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.845813 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845797 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.845954 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.846070 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845960 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") pod \"2a8fa477-4b50-46af-b0cf-fe5801946921\" (UID: \"2a8fa477-4b50-46af-b0cf-fe5801946921\") " Nov 02 14:58:32.846070 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.845980 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:32.846257 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.846225 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-trusted-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.846365 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.846267 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:32.848223 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.848200 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "kube-api-access-dsnm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:32.848332 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.848314 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:32.848477 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.848450 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:32.848538 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.848493 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:32.848538 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.848530 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:32.852806 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.852782 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted" pod "2a8fa477-4b50-46af-b0cf-fe5801946921" (UID: "2a8fa477-4b50-46af-b0cf-fe5801946921"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:32.946808 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946738 5091 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-certificates\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.946808 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946767 5091 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a8fa477-4b50-46af-b0cf-fe5801946921-ca-trust-extracted\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.946808 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946781 5091 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-registry-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.946808 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946796 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsnm7\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-kube-api-access-dsnm7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.947120 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946811 5091 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-image-registry-private-configuration\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.947120 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946825 5091 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a8fa477-4b50-46af-b0cf-fe5801946921-bound-sa-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:32.947120 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:32.946837 5091 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a8fa477-4b50-46af-b0cf-fe5801946921-installation-pull-secrets\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:33.412491 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.412455 5091 generic.go:356] "Generic (PLEG): container finished" podID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerID="33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331" exitCode=0 Nov 02 14:58:33.412686 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.412504 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" Nov 02 14:58:33.412686 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.412548 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" event={"ID":"2a8fa477-4b50-46af-b0cf-fe5801946921","Type":"ContainerDied","Data":"33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331"} Nov 02 14:58:33.412686 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.412606 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59c9b7fcd6-bxrc7" event={"ID":"2a8fa477-4b50-46af-b0cf-fe5801946921","Type":"ContainerDied","Data":"3b95f79424dd4ce418531338c47dbe6cc66a530f7b9ed17396afe95d0aea3141"} Nov 02 14:58:33.412686 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.412627 5091 scope.go:122] "RemoveContainer" containerID="33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331" Nov 02 14:58:33.423879 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.423686 5091 scope.go:122] "RemoveContainer" containerID="33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331" Nov 02 14:58:33.424122 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:33.423995 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331\": container with ID starting with 33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331 not found: ID does not exist" containerID="33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331" Nov 02 14:58:33.424122 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.424020 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331"} err="failed to get container status \"33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331\": rpc error: code = NotFound desc = could not find container \"33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331\": container with ID starting with 33d5b49e55206bbfc12a08b724f985d24f0abd28c8c2a60a319934c810158331 not found: ID does not exist" Nov 02 14:58:33.619121 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.619093 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59c9b7fcd6-bxrc7"] Nov 02 14:58:33.623394 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.623375 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59c9b7fcd6-bxrc7"] Nov 02 14:58:33.958831 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:33.958803 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" path="/var/lib/kubelet/pods/2a8fa477-4b50-46af-b0cf-fe5801946921/volumes" Nov 02 14:58:34.369156 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.369100 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5684f6796c-cxfkt" podUID="07703925-bba5-4744-82af-6c35e9d7e5fd" containerName="console" containerID="cri-o://f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f" gracePeriod=15 Nov 02 14:58:34.651230 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.651201 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:34.761552 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761499 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.761552 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761553 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq\" (UniqueName: \"kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.761775 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761594 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.761775 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761658 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.761775 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761683 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.761953 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761840 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.761953 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761911 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.762057 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761960 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert\") pod \"07703925-bba5-4744-82af-6c35e9d7e5fd\" (UID: \"07703925-bba5-4744-82af-6c35e9d7e5fd\") " Nov 02 14:58:34.762057 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.761879 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:34.762168 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762063 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:34.762168 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762068 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:34.762330 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762305 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:34.762497 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:34.762636 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762620 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.762789 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762644 5091 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07703925-bba5-4744-82af-6c35e9d7e5fd-tmp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.762896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762811 5091 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-oauth-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.762896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762827 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-trusted-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.762896 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.762840 5091 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/07703925-bba5-4744-82af-6c35e9d7e5fd-console-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.764294 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.764268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:34.764377 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.764287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:34.764377 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.764340 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq" pod "07703925-bba5-4744-82af-6c35e9d7e5fd" (UID: "07703925-bba5-4744-82af-6c35e9d7e5fd"). InnerVolumeSpecName "kube-api-access-vnwvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:34.864138 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.864107 5091 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-oauth-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.864138 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.864135 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwvq\" (UniqueName: \"kubernetes.io/projected/07703925-bba5-4744-82af-6c35e9d7e5fd-kube-api-access-vnwvq\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:34.864138 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:34.864145 5091 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/07703925-bba5-4744-82af-6c35e9d7e5fd-console-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:35.156988 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.156956 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5799fcb47f-gtk4f"] Nov 02 14:58:35.157320 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.157308 5091 cpu_manager.go:414] "containerMap: removing container" podUID="07703925-bba5-4744-82af-6c35e9d7e5fd" containerName="console" Nov 02 14:58:35.157371 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.157321 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07703925-bba5-4744-82af-6c35e9d7e5fd" containerName="console" Nov 02 14:58:35.157371 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.157330 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerName="registry" Nov 02 14:58:35.157371 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.157335 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerName="registry" Nov 02 14:58:35.157459 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.157409 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="07703925-bba5-4744-82af-6c35e9d7e5fd" containerName="console" Nov 02 14:58:35.157459 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.157416 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8fa477-4b50-46af-b0cf-fe5801946921" containerName="registry" Nov 02 14:58:35.188956 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.188931 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.255744 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.255713 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5799fcb47f-gtk4f"] Nov 02 14:58:35.266593 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-trusted-ca-bundle\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.266719 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266631 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-service-ca\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.266763 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266710 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-oauth-serving-cert\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.266763 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266755 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-console-config\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.266838 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fece4045-19f1-4a6a-a1c7-d335770281fc-console-serving-cert\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.266870 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266855 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpjpk\" (UniqueName: \"kubernetes.io/projected/fece4045-19f1-4a6a-a1c7-d335770281fc-kube-api-access-dpjpk\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.266903 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fece4045-19f1-4a6a-a1c7-d335770281fc-tmp\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.267001 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.266985 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fece4045-19f1-4a6a-a1c7-d335770281fc-console-oauth-config\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368079 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368045 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fece4045-19f1-4a6a-a1c7-d335770281fc-console-serving-cert\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368239 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368089 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpjpk\" (UniqueName: \"kubernetes.io/projected/fece4045-19f1-4a6a-a1c7-d335770281fc-kube-api-access-dpjpk\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368239 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368112 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fece4045-19f1-4a6a-a1c7-d335770281fc-tmp\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368239 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368138 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fece4045-19f1-4a6a-a1c7-d335770281fc-console-oauth-config\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368239 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368156 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-trusted-ca-bundle\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368239 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368194 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-service-ca\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.368976 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.368895 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fece4045-19f1-4a6a-a1c7-d335770281fc-tmp\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.371165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.369050 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-oauth-serving-cert\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.371165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.369275 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-service-ca\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.371165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.369278 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-console-config\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.371165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.369455 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-trusted-ca-bundle\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.371165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.369777 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-oauth-serving-cert\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.371165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.369891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fece4045-19f1-4a6a-a1c7-d335770281fc-console-config\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.374842 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.372016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fece4045-19f1-4a6a-a1c7-d335770281fc-console-oauth-config\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.374842 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.373841 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fece4045-19f1-4a6a-a1c7-d335770281fc-console-serving-cert\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.423371 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.423316 5091 generic.go:356] "Generic (PLEG): container finished" podID="07703925-bba5-4744-82af-6c35e9d7e5fd" containerID="f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f" exitCode=0 Nov 02 14:58:35.423371 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.423357 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5684f6796c-cxfkt" event={"ID":"07703925-bba5-4744-82af-6c35e9d7e5fd","Type":"ContainerDied","Data":"f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f"} Nov 02 14:58:35.423496 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.423373 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5684f6796c-cxfkt" Nov 02 14:58:35.423496 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.423426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5684f6796c-cxfkt" event={"ID":"07703925-bba5-4744-82af-6c35e9d7e5fd","Type":"ContainerDied","Data":"9a3d17ec665165e0907b316219bb4e03e7153fb145bec9fd3992f1eae302e2d2"} Nov 02 14:58:35.423496 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.423446 5091 scope.go:122] "RemoveContainer" containerID="f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f" Nov 02 14:58:35.427425 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.427402 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpjpk\" (UniqueName: \"kubernetes.io/projected/fece4045-19f1-4a6a-a1c7-d335770281fc-kube-api-access-dpjpk\") pod \"console-5799fcb47f-gtk4f\" (UID: \"fece4045-19f1-4a6a-a1c7-d335770281fc\") " pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.438375 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.438360 5091 scope.go:122] "RemoveContainer" containerID="f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f" Nov 02 14:58:35.438653 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:35.438634 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f\": container with ID starting with f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f not found: ID does not exist" containerID="f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f" Nov 02 14:58:35.438709 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.438659 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f"} err="failed to get container status \"f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f\": rpc error: code = NotFound desc = could not find container \"f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f\": container with ID starting with f1857b25830b05489e6ee2fbb799d08ff41895d28e38f570812ac2a508f5331f not found: ID does not exist" Nov 02 14:58:35.509483 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.509453 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:35.592473 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.592441 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5684f6796c-cxfkt"] Nov 02 14:58:35.594253 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.594223 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5684f6796c-cxfkt"] Nov 02 14:58:35.890590 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.890556 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5799fcb47f-gtk4f"] Nov 02 14:58:35.896839 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:58:35.896812 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfece4045_19f1_4a6a_a1c7_d335770281fc.slice/crio-be46b7c430e009aea624034f779b27d03ba16a40c16fa6dcf89a5324d6e6152a WatchSource:0}: Error finding container be46b7c430e009aea624034f779b27d03ba16a40c16fa6dcf89a5324d6e6152a: Status 404 returned error can't find the container with id be46b7c430e009aea624034f779b27d03ba16a40c16fa6dcf89a5324d6e6152a Nov 02 14:58:35.958657 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:35.958634 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07703925-bba5-4744-82af-6c35e9d7e5fd" path="/var/lib/kubelet/pods/07703925-bba5-4744-82af-6c35e9d7e5fd/volumes" Nov 02 14:58:36.429435 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:36.429391 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5799fcb47f-gtk4f" event={"ID":"fece4045-19f1-4a6a-a1c7-d335770281fc","Type":"ContainerStarted","Data":"1012c09367796dc2cc83944b53df942b87deb1c2a66a9775f9c6fbcb37b8939c"} Nov 02 14:58:36.429621 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:36.429447 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5799fcb47f-gtk4f" event={"ID":"fece4045-19f1-4a6a-a1c7-d335770281fc","Type":"ContainerStarted","Data":"be46b7c430e009aea624034f779b27d03ba16a40c16fa6dcf89a5324d6e6152a"} Nov 02 14:58:36.544322 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:36.544270 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5799fcb47f-gtk4f" podStartSLOduration=1.544257625 podStartE2EDuration="1.544257625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:36.543838963 +0000 UTC m=+70.708391504" watchObservedRunningTime="2026-11-02 14:58:36.544257625 +0000 UTC m=+70.708810164" Nov 02 14:58:38.797450 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:38.797401 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c4cd949c9-85zr4" podUID="a084235b-a3c7-4215-ae03-cd225c30f1b9" containerName="console" containerID="cri-o://9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240" gracePeriod=15 Nov 02 14:58:39.089421 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.089395 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:39.202034 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202001 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202186 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202063 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202186 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202100 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202186 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202138 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202186 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202180 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202413 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202222 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202471 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202422 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202544 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202440 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:39.202544 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202479 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz\" (UniqueName: \"kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz\") pod \"a084235b-a3c7-4215-ae03-cd225c30f1b9\" (UID: \"a084235b-a3c7-4215-ae03-cd225c30f1b9\") " Nov 02 14:58:39.202544 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202501 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:39.202699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202576 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:39.202699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:39.202888 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202861 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:39.202947 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202895 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-trusted-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.202947 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202917 5091 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.202947 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202931 5091 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a084235b-a3c7-4215-ae03-cd225c30f1b9-tmp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.202947 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.202944 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.204789 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.204766 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:39.205154 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.205123 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:39.205293 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.205246 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz" pod "a084235b-a3c7-4215-ae03-cd225c30f1b9" (UID: "a084235b-a3c7-4215-ae03-cd225c30f1b9"). InnerVolumeSpecName "kube-api-access-xcclz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:39.303694 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.303617 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcclz\" (UniqueName: \"kubernetes.io/projected/a084235b-a3c7-4215-ae03-cd225c30f1b9-kube-api-access-xcclz\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.303694 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.303645 5091 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-oauth-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.303694 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.303658 5091 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a084235b-a3c7-4215-ae03-cd225c30f1b9-console-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.303694 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.303670 5091 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a084235b-a3c7-4215-ae03-cd225c30f1b9-oauth-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:39.445527 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.445487 5091 generic.go:356] "Generic (PLEG): container finished" podID="a084235b-a3c7-4215-ae03-cd225c30f1b9" containerID="9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240" exitCode=0 Nov 02 14:58:39.445699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.445543 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cd949c9-85zr4" event={"ID":"a084235b-a3c7-4215-ae03-cd225c30f1b9","Type":"ContainerDied","Data":"9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240"} Nov 02 14:58:39.445699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.445560 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4cd949c9-85zr4" Nov 02 14:58:39.445699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.445585 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4cd949c9-85zr4" event={"ID":"a084235b-a3c7-4215-ae03-cd225c30f1b9","Type":"ContainerDied","Data":"076768219f44b7d1627943333e7cc07e18e96746c92b0fcfb1c2dc20c180481d"} Nov 02 14:58:39.445699 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.445607 5091 scope.go:122] "RemoveContainer" containerID="9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240" Nov 02 14:58:39.454487 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.454468 5091 scope.go:122] "RemoveContainer" containerID="9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240" Nov 02 14:58:39.454759 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:39.454735 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240\": container with ID starting with 9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240 not found: ID does not exist" containerID="9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240" Nov 02 14:58:39.454849 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.454766 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240"} err="failed to get container status \"9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240\": rpc error: code = NotFound desc = could not find container \"9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240\": container with ID starting with 9c2b93b5ae9281aefc2cbf4cea37ae98b08b306ba671851c7b577aa03ff50240 not found: ID does not exist" Nov 02 14:58:39.610582 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.610497 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4cd949c9-85zr4"] Nov 02 14:58:39.657484 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.657455 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4cd949c9-85zr4"] Nov 02 14:58:39.957976 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:39.957947 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a084235b-a3c7-4215-ae03-cd225c30f1b9" path="/var/lib/kubelet/pods/a084235b-a3c7-4215-ae03-cd225c30f1b9/volumes" Nov 02 14:58:45.510343 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:45.510313 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:45.510723 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:45.510361 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:45.514871 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:45.514849 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:46.477064 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:46.477029 5091 generic.go:356] "Generic (PLEG): container finished" podID="66c2abab-7f39-4d34-8571-ad883dde9532" containerID="bc03f3ffa4100f6d7374261a489ab8bb128c4730fd6aa503c893f239b1669d85" exitCode=0 Nov 02 14:58:46.477278 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:46.477111 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" event={"ID":"66c2abab-7f39-4d34-8571-ad883dde9532","Type":"ContainerDied","Data":"bc03f3ffa4100f6d7374261a489ab8bb128c4730fd6aa503c893f239b1669d85"} Nov 02 14:58:46.477576 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:46.477554 5091 scope.go:122] "RemoveContainer" containerID="bc03f3ffa4100f6d7374261a489ab8bb128c4730fd6aa503c893f239b1669d85" Nov 02 14:58:46.481147 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:46.481126 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5799fcb47f-gtk4f" Nov 02 14:58:46.749355 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:46.749321 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f7bbb9-6xxvm"] Nov 02 14:58:47.414277 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.414241 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2nf2p-5rnb4"] Nov 02 14:58:47.414469 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.414433 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" podUID="60ef143c-9312-4bc0-8180-7864b729d2b2" containerName="insights-gathering" containerID="cri-o://edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a" gracePeriod=30 Nov 02 14:58:47.482778 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.482750 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-r6gt7" event={"ID":"66c2abab-7f39-4d34-8571-ad883dde9532","Type":"ContainerStarted","Data":"b86bf7188f7377b448063236084ec1aebc910839af77b5a461d4138387ac9a9d"} Nov 02 14:58:47.707734 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.707708 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:58:47.819824 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.819790 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-44zwb-ncplk"] Nov 02 14:58:47.820158 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.820122 5091 cpu_manager.go:414] "containerMap: removing container" podUID="60ef143c-9312-4bc0-8180-7864b729d2b2" containerName="insights-gathering" Nov 02 14:58:47.820158 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.820137 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ef143c-9312-4bc0-8180-7864b729d2b2" containerName="insights-gathering" Nov 02 14:58:47.820230 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.820160 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a084235b-a3c7-4215-ae03-cd225c30f1b9" containerName="console" Nov 02 14:58:47.820230 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.820165 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a084235b-a3c7-4215-ae03-cd225c30f1b9" containerName="console" Nov 02 14:58:47.820230 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.820226 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ef143c-9312-4bc0-8180-7864b729d2b2" containerName="insights-gathering" Nov 02 14:58:47.820315 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.820238 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a084235b-a3c7-4215-ae03-cd225c30f1b9" containerName="console" Nov 02 14:58:47.840544 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.840510 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:47.871067 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871046 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") pod \"60ef143c-9312-4bc0-8180-7864b729d2b2\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " Nov 02 14:58:47.871165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871084 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") pod \"60ef143c-9312-4bc0-8180-7864b729d2b2\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " Nov 02 14:58:47.871165 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871136 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") pod \"60ef143c-9312-4bc0-8180-7864b729d2b2\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " Nov 02 14:58:47.871284 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871191 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") pod \"60ef143c-9312-4bc0-8180-7864b729d2b2\" (UID: \"60ef143c-9312-4bc0-8180-7864b729d2b2\") " Nov 02 14:58:47.871472 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle" pod "60ef143c-9312-4bc0-8180-7864b729d2b2" (UID: "60ef143c-9312-4bc0-8180-7864b729d2b2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:47.871552 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871476 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle" pod "60ef143c-9312-4bc0-8180-7864b729d2b2" (UID: "60ef143c-9312-4bc0-8180-7864b729d2b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:47.871552 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.871538 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path" pod "60ef143c-9312-4bc0-8180-7864b729d2b2" (UID: "60ef143c-9312-4bc0-8180-7864b729d2b2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:47.873290 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.873268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp" pod "60ef143c-9312-4bc0-8180-7864b729d2b2" (UID: "60ef143c-9312-4bc0-8180-7864b729d2b2"). InnerVolumeSpecName "kube-api-access-68ttp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:47.878301 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.878279 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-44zwb-ncplk"] Nov 02 14:58:47.972169 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972145 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:47.972283 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972183 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:47.972325 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:47.972359 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972337 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stspr\" (UniqueName: \"kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:47.972406 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972394 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-service-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:47.972442 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972410 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/60ef143c-9312-4bc0-8180-7864b729d2b2-archives-path\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:47.972442 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972420 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68ttp\" (UniqueName: \"kubernetes.io/projected/60ef143c-9312-4bc0-8180-7864b729d2b2-kube-api-access-68ttp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:47.972506 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:47.972457 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ef143c-9312-4bc0-8180-7864b729d2b2-trusted-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:48.073558 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.073502 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.073715 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.073568 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stspr\" (UniqueName: \"kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.073715 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.073597 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.073715 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.073626 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.074030 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.074010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.074194 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.074174 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.074255 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.074236 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.132840 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.132818 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stspr\" (UniqueName: \"kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr\") pod \"periodic-gathering-44zwb-ncplk\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.168744 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.168715 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 14:58:48.488965 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.488882 5091 generic.go:356] "Generic (PLEG): container finished" podID="60ef143c-9312-4bc0-8180-7864b729d2b2" containerID="edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a" exitCode=2 Nov 02 14:58:48.488965 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.488935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" event={"ID":"60ef143c-9312-4bc0-8180-7864b729d2b2","Type":"ContainerDied","Data":"edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a"} Nov 02 14:58:48.488965 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.488948 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" Nov 02 14:58:48.489221 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.488974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2nf2p-5rnb4" event={"ID":"60ef143c-9312-4bc0-8180-7864b729d2b2","Type":"ContainerDied","Data":"d955faf7c07f6e55896ce45bacdd725d9f657f7bf8617a7d5203333408712c22"} Nov 02 14:58:48.489221 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.489000 5091 scope.go:122] "RemoveContainer" containerID="edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a" Nov 02 14:58:48.497159 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.497127 5091 scope.go:122] "RemoveContainer" containerID="edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a" Nov 02 14:58:48.497408 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:48.497388 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a\": container with ID starting with edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a not found: ID does not exist" containerID="edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a" Nov 02 14:58:48.497473 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.497421 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a"} err="failed to get container status \"edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a\": rpc error: code = NotFound desc = could not find container \"edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a\": container with ID starting with edcc37c1c8b438c9fa2be0512fd58552317bc5caf20ca901b5f646ad4455d50a not found: ID does not exist" Nov 02 14:58:48.555485 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.555454 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-44zwb-ncplk"] Nov 02 14:58:48.561326 ip-10-0-130-229 kubenswrapper[5091]: W1102 14:58:48.561295 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2758be0_bbcc_4ea8_b289_6fef0c533293.slice/crio-4ca34b57d787f039b6aa7f18da538ffba04dc41f81058dd8ce590df0a6a15413 WatchSource:0}: Error finding container 4ca34b57d787f039b6aa7f18da538ffba04dc41f81058dd8ce590df0a6a15413: Status 404 returned error can't find the container with id 4ca34b57d787f039b6aa7f18da538ffba04dc41f81058dd8ce590df0a6a15413 Nov 02 14:58:48.646606 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.646579 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2nf2p-5rnb4"] Nov 02 14:58:48.649087 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:48.649064 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-2nf2p-5rnb4"] Nov 02 14:58:49.496446 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.496411 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44zwb-ncplk" event={"ID":"e2758be0-bbcc-4ea8-b289-6fef0c533293","Type":"ContainerStarted","Data":"36f4726e9648c041eb3ea431aa8829e443c4877d6fc2b7e0c6548b23e4f9a156"} Nov 02 14:58:49.496961 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.496466 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44zwb-ncplk" event={"ID":"e2758be0-bbcc-4ea8-b289-6fef0c533293","Type":"ContainerStarted","Data":"4ca34b57d787f039b6aa7f18da538ffba04dc41f81058dd8ce590df0a6a15413"} Nov 02 14:58:49.550551 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.550503 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-76868ccd4b-zltbl" podUID="3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" containerName="console" containerID="cri-o://dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84" gracePeriod=15 Nov 02 14:58:49.607646 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.607597 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-44zwb-ncplk" podStartSLOduration=2.607583769 podStartE2EDuration="2.607583769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 14:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 14:58:49.607303855 +0000 UTC m=+83.771856394" watchObservedRunningTime="2026-11-02 14:58:49.607583769 +0000 UTC m=+83.772136309" Nov 02 14:58:49.852118 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.852091 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:58:49.958125 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.958095 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ef143c-9312-4bc0-8180-7864b729d2b2" path="/var/lib/kubelet/pods/60ef143c-9312-4bc0-8180-7864b729d2b2/volumes" Nov 02 14:58:49.988652 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988623 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv\" (UniqueName: \"kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.988837 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988699 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.988837 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988747 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.988837 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988773 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.988837 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988797 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.988837 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988823 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.989116 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.988843 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp\") pod \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\" (UID: \"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1\") " Nov 02 14:58:49.989219 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.989190 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:58:49.989670 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.989628 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.990667 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.990632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.990928 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.990905 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:58:49.991871 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.991723 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:49.991871 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.991823 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "kube-api-access-fm9dv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:58:49.992021 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:49.991932 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert" pod "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" (UID: "3d3dc6a4-59c3-4d5b-af4c-5e94685902d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:58:50.089761 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089685 5091 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.089761 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089712 5091 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.089761 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089725 5091 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-console-oauth-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.089761 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089739 5091 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-oauth-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.089761 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089752 5091 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-tmp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.090014 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089767 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm9dv\" (UniqueName: \"kubernetes.io/projected/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-kube-api-access-fm9dv\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.090014 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.089781 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:58:50.501835 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.501800 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" containerID="dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84" exitCode=0 Nov 02 14:58:50.502221 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.501844 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76868ccd4b-zltbl" Nov 02 14:58:50.502221 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.501880 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76868ccd4b-zltbl" event={"ID":"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1","Type":"ContainerDied","Data":"dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84"} Nov 02 14:58:50.502221 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.501934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76868ccd4b-zltbl" event={"ID":"3d3dc6a4-59c3-4d5b-af4c-5e94685902d1","Type":"ContainerDied","Data":"0bc20bf581f66f782275517d00bf8ee4d4ce5c523921e31bb163603543c39c6d"} Nov 02 14:58:50.502221 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.501960 5091 scope.go:122] "RemoveContainer" containerID="dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84" Nov 02 14:58:50.511269 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.511240 5091 scope.go:122] "RemoveContainer" containerID="dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84" Nov 02 14:58:50.511539 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:50.511501 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84\": container with ID starting with dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84 not found: ID does not exist" containerID="dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84" Nov 02 14:58:50.511616 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.511549 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84"} err="failed to get container status \"dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84\": rpc error: code = NotFound desc = could not find container \"dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84\": container with ID starting with dcb3266832bfbe9386c44fe4b8ceeaacbcb90e504c4c8ee55270938a3f4eda84 not found: ID does not exist" Nov 02 14:58:50.658878 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.658845 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76868ccd4b-zltbl"] Nov 02 14:58:50.706018 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:50.705981 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76868ccd4b-zltbl"] Nov 02 14:58:51.958021 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:51.957989 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" path="/var/lib/kubelet/pods/3d3dc6a4-59c3-4d5b-af4c-5e94685902d1/volumes" Nov 02 14:58:56.528025 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:56.527992 5091 generic.go:356] "Generic (PLEG): container finished" podID="cc4a9361-8679-4e65-83f4-d8d876321372" containerID="971f2ed6513fdfd366b7ff40b6ba7a9302f80dc8cf6a0a1bd61b42b6240bc9fd" exitCode=0 Nov 02 14:58:56.528560 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:56.528068 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" event={"ID":"cc4a9361-8679-4e65-83f4-d8d876321372","Type":"ContainerDied","Data":"971f2ed6513fdfd366b7ff40b6ba7a9302f80dc8cf6a0a1bd61b42b6240bc9fd"} Nov 02 14:58:56.528560 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:56.528499 5091 scope.go:122] "RemoveContainer" containerID="971f2ed6513fdfd366b7ff40b6ba7a9302f80dc8cf6a0a1bd61b42b6240bc9fd" Nov 02 14:58:57.535473 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:58:57.535437 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-t6d8f" event={"ID":"cc4a9361-8679-4e65-83f4-d8d876321372","Type":"ContainerStarted","Data":"2d2a52146f342e5748fc5fe2e9c8dd5de137ac3a8c0928090e73e342837fed7d"} Nov 02 14:58:57.933788 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:58:57.933748 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 14:59:11.772940 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:11.772878 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-75f7bbb9-6xxvm" podUID="2fb3a1ce-8c03-4418-8d09-390269d024d7" containerName="console" containerID="cri-o://407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e" gracePeriod=15 Nov 02 14:59:12.039935 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.039911 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:59:12.074069 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074043 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074173 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074083 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074173 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074104 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074173 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074135 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074328 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074181 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074328 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074209 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp\" (UniqueName: \"kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074433 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074352 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074486 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074435 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config\") pod \"2fb3a1ce-8c03-4418-8d09-390269d024d7\" (UID: \"2fb3a1ce-8c03-4418-8d09-390269d024d7\") " Nov 02 14:59:12.074597 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074572 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:12.074666 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074636 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:12.074719 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074678 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:12.074774 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074755 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 14:59:12.074909 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074888 5091 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-oauth-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.074967 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074913 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-trusted-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.074967 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074928 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.074967 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074943 5091 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2fb3a1ce-8c03-4418-8d09-390269d024d7-tmp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.074967 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.074950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 14:59:12.076298 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.076272 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:59:12.076746 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.076725 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 14:59:12.076746 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.076738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp" pod "2fb3a1ce-8c03-4418-8d09-390269d024d7" (UID: "2fb3a1ce-8c03-4418-8d09-390269d024d7"). InnerVolumeSpecName "kube-api-access-nr5gp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 14:59:12.176155 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.176122 5091 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-serving-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.176155 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.176146 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr5gp\" (UniqueName: \"kubernetes.io/projected/2fb3a1ce-8c03-4418-8d09-390269d024d7-kube-api-access-nr5gp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.176155 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.176156 5091 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.176402 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.176165 5091 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2fb3a1ce-8c03-4418-8d09-390269d024d7-console-oauth-config\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 14:59:12.600968 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.600930 5091 generic.go:356] "Generic (PLEG): container finished" podID="2fb3a1ce-8c03-4418-8d09-390269d024d7" containerID="407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e" exitCode=0 Nov 02 14:59:12.601174 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.600985 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f7bbb9-6xxvm" Nov 02 14:59:12.601174 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.601002 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f7bbb9-6xxvm" event={"ID":"2fb3a1ce-8c03-4418-8d09-390269d024d7","Type":"ContainerDied","Data":"407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e"} Nov 02 14:59:12.601174 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.601043 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f7bbb9-6xxvm" event={"ID":"2fb3a1ce-8c03-4418-8d09-390269d024d7","Type":"ContainerDied","Data":"ff57ab1adeb74e50240dca66b85c32fb26fc70df01e355a12628512d655d94e0"} Nov 02 14:59:12.601174 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.601062 5091 scope.go:122] "RemoveContainer" containerID="407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e" Nov 02 14:59:12.610263 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.610247 5091 scope.go:122] "RemoveContainer" containerID="407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e" Nov 02 14:59:12.610576 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:59:12.610558 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e\": container with ID starting with 407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e not found: ID does not exist" containerID="407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e" Nov 02 14:59:12.610621 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.610585 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e"} err="failed to get container status \"407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e\": rpc error: code = NotFound desc = could not find container \"407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e\": container with ID starting with 407c72d37aeeeaa1f3303957b1022304772e34c03afa5caa330e813aa560bc6e not found: ID does not exist" Nov 02 14:59:12.783087 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.783059 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f7bbb9-6xxvm"] Nov 02 14:59:12.829735 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:12.829708 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f7bbb9-6xxvm"] Nov 02 14:59:13.958448 ip-10-0-130-229 kubenswrapper[5091]: I1102 14:59:13.958417 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fb3a1ce-8c03-4418-8d09-390269d024d7" path="/var/lib/kubelet/pods/2fb3a1ce-8c03-4418-8d09-390269d024d7/volumes" Nov 02 14:59:27.933132 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:59:27.933094 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 14:59:57.932884 ip-10-0-130-229 kubenswrapper[5091]: E1102 14:59:57.932798 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:00:10.846158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:10.846120 5091 generic.go:356] "Generic (PLEG): container finished" podID="e2758be0-bbcc-4ea8-b289-6fef0c533293" containerID="36f4726e9648c041eb3ea431aa8829e443c4877d6fc2b7e0c6548b23e4f9a156" exitCode=0 Nov 02 15:00:10.846640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:10.846195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44zwb-ncplk" event={"ID":"e2758be0-bbcc-4ea8-b289-6fef0c533293","Type":"ContainerDied","Data":"36f4726e9648c041eb3ea431aa8829e443c4877d6fc2b7e0c6548b23e4f9a156"} Nov 02 15:00:11.984041 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:11.984016 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 15:00:12.032237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path\") pod \"e2758be0-bbcc-4ea8-b289-6fef0c533293\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " Nov 02 15:00:12.032372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032286 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle\") pod \"e2758be0-bbcc-4ea8-b289-6fef0c533293\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " Nov 02 15:00:12.032372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr\" (UniqueName: \"kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr\") pod \"e2758be0-bbcc-4ea8-b289-6fef0c533293\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " Nov 02 15:00:12.032480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032374 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle\") pod \"e2758be0-bbcc-4ea8-b289-6fef0c533293\" (UID: \"e2758be0-bbcc-4ea8-b289-6fef0c533293\") " Nov 02 15:00:12.032610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032559 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path" pod "e2758be0-bbcc-4ea8-b289-6fef0c533293" (UID: "e2758be0-bbcc-4ea8-b289-6fef0c533293"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:00:12.032663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032635 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle" pod "e2758be0-bbcc-4ea8-b289-6fef0c533293" (UID: "e2758be0-bbcc-4ea8-b289-6fef0c533293"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:00:12.032754 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032738 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/e2758be0-bbcc-4ea8-b289-6fef0c533293-archives-path\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:00:12.032810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.032794 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle" pod "e2758be0-bbcc-4ea8-b289-6fef0c533293" (UID: "e2758be0-bbcc-4ea8-b289-6fef0c533293"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:00:12.034356 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.034333 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr" pod "e2758be0-bbcc-4ea8-b289-6fef0c533293" (UID: "e2758be0-bbcc-4ea8-b289-6fef0c533293"). InnerVolumeSpecName "kube-api-access-stspr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:00:12.133994 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.133964 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-service-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:00:12.134147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.133995 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stspr\" (UniqueName: \"kubernetes.io/projected/e2758be0-bbcc-4ea8-b289-6fef0c533293-kube-api-access-stspr\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:00:12.134147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.134010 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2758be0-bbcc-4ea8-b289-6fef0c533293-trusted-ca-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:00:12.855396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.855370 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-44zwb-ncplk" Nov 02 15:00:12.855603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.855369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-44zwb-ncplk" event={"ID":"e2758be0-bbcc-4ea8-b289-6fef0c533293","Type":"ContainerDied","Data":"4ca34b57d787f039b6aa7f18da538ffba04dc41f81058dd8ce590df0a6a15413"} Nov 02 15:00:12.855603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:00:12.855493 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ca34b57d787f039b6aa7f18da538ffba04dc41f81058dd8ce590df0a6a15413" Nov 02 15:00:27.933094 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:00:27.933055 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:00:57.933463 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:00:57.933419 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:01:27.933359 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:01:27.933323 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:01:57.932952 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:01:57.932911 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:02:25.972028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:02:25.972001 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:02:25.974488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:02:25.974467 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:02:27.932660 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:02:27.932624 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:02:57.932680 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:02:57.932603 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:03:27.932916 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:03:27.932873 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:03:57.933627 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:03:57.933586 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:04:27.933687 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:04:27.933649 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:04:28.454723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.454692 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l"] Nov 02 15:04:28.455018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455006 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2fb3a1ce-8c03-4418-8d09-390269d024d7" containerName="console" Nov 02 15:04:28.455069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455019 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fb3a1ce-8c03-4418-8d09-390269d024d7" containerName="console" Nov 02 15:04:28.455069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455029 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e2758be0-bbcc-4ea8-b289-6fef0c533293" containerName="insights-gathering" Nov 02 15:04:28.455069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455034 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2758be0-bbcc-4ea8-b289-6fef0c533293" containerName="insights-gathering" Nov 02 15:04:28.455069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455051 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" containerName="console" Nov 02 15:04:28.455069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455057 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" containerName="console" Nov 02 15:04:28.455218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455113 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fb3a1ce-8c03-4418-8d09-390269d024d7" containerName="console" Nov 02 15:04:28.455218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455119 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2758be0-bbcc-4ea8-b289-6fef0c533293" containerName="insights-gathering" Nov 02 15:04:28.455218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.455126 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3dc6a4-59c3-4d5b-af4c-5e94685902d1" containerName="console" Nov 02 15:04:28.458199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.458181 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.508341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.508311 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 15:04:28.555571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.555542 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 15:04:28.555687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.555600 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-4bxn5\"" Nov 02 15:04:28.556833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.556815 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l"] Nov 02 15:04:28.638586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.638563 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.638724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.638600 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjdz8\" (UniqueName: \"kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.638724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.638688 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.739929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.739859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.739929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.739901 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjdz8\" (UniqueName: \"kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.740091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.739963 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.740347 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.740325 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.740385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.740332 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:28.798843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:28.798811 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjdz8\" (UniqueName: \"kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:29.073176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:29.073102 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:29.472531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:29.472495 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l"] Nov 02 15:04:29.476772 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:04:29.476745 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeffa60b1_7454_42b9_93b3_cdbf60e6c782.slice/crio-b957b7a9d864840ba78b2ad5c89817477e3e820dffd7e69050246637f1f09bc1 WatchSource:0}: Error finding container b957b7a9d864840ba78b2ad5c89817477e3e820dffd7e69050246637f1f09bc1: Status 404 returned error can't find the container with id b957b7a9d864840ba78b2ad5c89817477e3e820dffd7e69050246637f1f09bc1 Nov 02 15:04:29.479066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:29.479048 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:04:29.883198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:29.883154 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" event={"ID":"effa60b1-7454-42b9-93b3-cdbf60e6c782","Type":"ContainerStarted","Data":"b957b7a9d864840ba78b2ad5c89817477e3e820dffd7e69050246637f1f09bc1"} Nov 02 15:04:30.724462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.724423 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9"] Nov 02 15:04:30.728377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.728350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.778609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.778510 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 15:04:30.799917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.799885 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x6s8k\"" Nov 02 15:04:30.827086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.827056 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9"] Nov 02 15:04:30.854926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.854895 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.854932 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.854962 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.855081 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.855108 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.855137 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.855249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.855268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.855314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.855293 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qb8q\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956587 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956655 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956689 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956789 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956820 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956857 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956873 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.956958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.956895 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qb8q\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.957208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.957022 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.957501 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.957475 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.957614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.957547 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.957798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.957760 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.957956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.957932 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.959387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.959368 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:30.959745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:30.959722 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:31.026818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.026747 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:31.032193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.032162 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qb8q\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-k7pr9\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:31.043540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.043484 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:31.479766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.479728 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9"] Nov 02 15:04:31.484608 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:04:31.484499 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a5dc38_af8a_4235_9d70_ed61bb284a19.slice/crio-631860e3c378e860c7ece08bf6580289cf181f7e863376973fe1474c9be4284a WatchSource:0}: Error finding container 631860e3c378e860c7ece08bf6580289cf181f7e863376973fe1474c9be4284a: Status 404 returned error can't find the container with id 631860e3c378e860c7ece08bf6580289cf181f7e863376973fe1474c9be4284a Nov 02 15:04:31.726673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.726636 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2"] Nov 02 15:04:31.731864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.731806 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.765024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.764991 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb"] Nov 02 15:04:31.768357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.768337 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.783679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.783652 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-zpxjt\"" Nov 02 15:04:31.785065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.785042 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2"] Nov 02 15:04:31.823841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.823819 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb"] Nov 02 15:04:31.864504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.864471 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mrtf\" (UniqueName: \"kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.864633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.864544 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.864633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.864616 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.864736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.864646 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.864736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.864689 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.864901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.864880 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csb5t\" (UniqueName: \"kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.883853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.883827 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-h4dp6\"" Nov 02 15:04:31.894662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.894629 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" event={"ID":"17a5dc38-af8a-4235-9d70-ed61bb284a19","Type":"ContainerStarted","Data":"631860e3c378e860c7ece08bf6580289cf181f7e863376973fe1474c9be4284a"} Nov 02 15:04:31.965857 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.965829 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.966019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.965931 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csb5t\" (UniqueName: \"kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.966019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.965979 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mrtf\" (UniqueName: \"kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.966019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966005 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.966183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966031 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.966183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.966288 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:31.966441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966422 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.966536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966472 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:31.966729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:31.966710 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:32.025271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.025206 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mrtf\" (UniqueName: \"kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:32.025271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.025204 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csb5t\" (UniqueName: \"kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:32.051044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.051010 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:32.083457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.083430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:32.437981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.437949 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2"] Nov 02 15:04:32.443934 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:04:32.443901 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b64fc7_406f_48ef_8211_4ecec653e3c7.slice/crio-9e011cd7a004be5d664ac100b1a487bb69b90ef8f3d150b3228db8e6d4562b7b WatchSource:0}: Error finding container 9e011cd7a004be5d664ac100b1a487bb69b90ef8f3d150b3228db8e6d4562b7b: Status 404 returned error can't find the container with id 9e011cd7a004be5d664ac100b1a487bb69b90ef8f3d150b3228db8e6d4562b7b Nov 02 15:04:32.488033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.488008 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb"] Nov 02 15:04:32.493878 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:04:32.493851 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff521e8_c4d3_40a9_a9fa_ff3290d07556.slice/crio-eecda5b92056174d291679009f422f5998cebbced05537587307e45debbdb9d8 WatchSource:0}: Error finding container eecda5b92056174d291679009f422f5998cebbced05537587307e45debbdb9d8: Status 404 returned error can't find the container with id eecda5b92056174d291679009f422f5998cebbced05537587307e45debbdb9d8 Nov 02 15:04:32.901167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.901130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" event={"ID":"b4b64fc7-406f-48ef-8211-4ecec653e3c7","Type":"ContainerStarted","Data":"9e011cd7a004be5d664ac100b1a487bb69b90ef8f3d150b3228db8e6d4562b7b"} Nov 02 15:04:32.902388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:32.902358 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" event={"ID":"0ff521e8-c4d3-40a9-a9fa-ff3290d07556","Type":"ContainerStarted","Data":"eecda5b92056174d291679009f422f5998cebbced05537587307e45debbdb9d8"} Nov 02 15:04:36.958896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:36.958623 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:36.958896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:36.958708 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:36.958896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:36.958746 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:04:37.944013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:37.943378 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" event={"ID":"17a5dc38-af8a-4235-9d70-ed61bb284a19","Type":"ContainerStarted","Data":"3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883"} Nov 02 15:04:37.945566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:37.945534 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" event={"ID":"effa60b1-7454-42b9-93b3-cdbf60e6c782","Type":"ContainerStarted","Data":"3785a18e285ceffdbcc36ed3e7bface75f6bd030dcc9fe612311e3c6f5e6357f"} Nov 02 15:04:38.044709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.044680 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:38.049970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.049948 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:38.089597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.089336 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podStartSLOduration=2.618379939 podStartE2EDuration="8.089321749s" totalImagesPullingTime="5.47094181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:38.087777994 +0000 UTC m=+432.252330536" watchObservedRunningTime="2026-11-02 15:04:38.089321749 +0000 UTC m=+432.253874289" Nov 02 15:04:38.951232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.951144 5091 generic.go:356] "Generic (PLEG): container finished" podID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerID="571326569b55f1dda3a9266b2a5f598e68f275e415f4121f1211f42d744db411" exitCode=0 Nov 02 15:04:38.951397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.951224 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" event={"ID":"b4b64fc7-406f-48ef-8211-4ecec653e3c7","Type":"ContainerDied","Data":"571326569b55f1dda3a9266b2a5f598e68f275e415f4121f1211f42d744db411"} Nov 02 15:04:38.952945 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.952923 5091 generic.go:356] "Generic (PLEG): container finished" podID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerID="3785a18e285ceffdbcc36ed3e7bface75f6bd030dcc9fe612311e3c6f5e6357f" exitCode=0 Nov 02 15:04:38.953063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.953014 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" event={"ID":"effa60b1-7454-42b9-93b3-cdbf60e6c782","Type":"ContainerDied","Data":"3785a18e285ceffdbcc36ed3e7bface75f6bd030dcc9fe612311e3c6f5e6357f"} Nov 02 15:04:38.954632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.954609 5091 generic.go:356] "Generic (PLEG): container finished" podID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerID="335ba0ac51f3c8eb47c25c2b3e31ec87a3ca69b5fadbfa3af6339b4ae7221847" exitCode=0 Nov 02 15:04:38.954789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.954670 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" event={"ID":"0ff521e8-c4d3-40a9-a9fa-ff3290d07556","Type":"ContainerDied","Data":"335ba0ac51f3c8eb47c25c2b3e31ec87a3ca69b5fadbfa3af6339b4ae7221847"} Nov 02 15:04:38.954949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.954926 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:38.955921 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:38.955906 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:04:41.969455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:41.969422 5091 generic.go:356] "Generic (PLEG): container finished" podID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerID="bc778ff201f71606458a6fac26b136522d1f27cfb10c092a1b35fb4aaf84bd63" exitCode=0 Nov 02 15:04:41.969845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:41.969500 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" event={"ID":"0ff521e8-c4d3-40a9-a9fa-ff3290d07556","Type":"ContainerDied","Data":"bc778ff201f71606458a6fac26b136522d1f27cfb10c092a1b35fb4aaf84bd63"} Nov 02 15:04:42.975741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:42.975673 5091 generic.go:356] "Generic (PLEG): container finished" podID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerID="13f7cbb7bcb96ad72ea10b2388b2fc0ccca2839f9b016a0932d4642b47e62335" exitCode=0 Nov 02 15:04:42.976189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:42.975763 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" event={"ID":"b4b64fc7-406f-48ef-8211-4ecec653e3c7","Type":"ContainerDied","Data":"13f7cbb7bcb96ad72ea10b2388b2fc0ccca2839f9b016a0932d4642b47e62335"} Nov 02 15:04:42.977794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:42.977681 5091 generic.go:356] "Generic (PLEG): container finished" podID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerID="9c6b257e437a1ed17d864474ba9e5e8a31668f7fd17aeda7b55faa1d870fec8b" exitCode=0 Nov 02 15:04:42.977794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:42.977763 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" event={"ID":"effa60b1-7454-42b9-93b3-cdbf60e6c782","Type":"ContainerDied","Data":"9c6b257e437a1ed17d864474ba9e5e8a31668f7fd17aeda7b55faa1d870fec8b"} Nov 02 15:04:51.028241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:51.028206 5091 generic.go:356] "Generic (PLEG): container finished" podID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerID="ce65673de9d63def96d5b6dffad02d078f9b57d30b8a362ca196f5ee378a543b" exitCode=0 Nov 02 15:04:51.028665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:51.028298 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" event={"ID":"0ff521e8-c4d3-40a9-a9fa-ff3290d07556","Type":"ContainerDied","Data":"ce65673de9d63def96d5b6dffad02d078f9b57d30b8a362ca196f5ee378a543b"} Nov 02 15:04:52.035362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.035275 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" event={"ID":"b4b64fc7-406f-48ef-8211-4ecec653e3c7","Type":"ContainerStarted","Data":"39bc1e2390077d5ea6e6b67580538f0d674037d52cf36b22e57c1b0dfe5bc85a"} Nov 02 15:04:52.037345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.037319 5091 generic.go:356] "Generic (PLEG): container finished" podID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerID="59618c397b8a5fe7adfee2343c819e8b19466e876c76f0b21f17fea156bfb7ee" exitCode=0 Nov 02 15:04:52.037469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.037391 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" event={"ID":"effa60b1-7454-42b9-93b3-cdbf60e6c782","Type":"ContainerDied","Data":"59618c397b8a5fe7adfee2343c819e8b19466e876c76f0b21f17fea156bfb7ee"} Nov 02 15:04:52.148896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.148851 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" podStartSLOduration=2.97450344 podStartE2EDuration="21.148834246s" totalImagesPullingTime="18.174330806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:04:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:04:52.148138559 +0000 UTC m=+446.312691100" watchObservedRunningTime="2026-11-02 15:04:52.148834246 +0000 UTC m=+446.313386785" Nov 02 15:04:52.214050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.214024 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:52.273291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.273261 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle\") pod \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " Nov 02 15:04:52.273438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.273301 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util\") pod \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " Nov 02 15:04:52.273438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.273332 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t\" (UniqueName: \"kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t\") pod \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\" (UID: \"0ff521e8-c4d3-40a9-a9fa-ff3290d07556\") " Nov 02 15:04:52.273758 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.273737 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle" pod "0ff521e8-c4d3-40a9-a9fa-ff3290d07556" (UID: "0ff521e8-c4d3-40a9-a9fa-ff3290d07556"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:52.275330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.275309 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t" pod "0ff521e8-c4d3-40a9-a9fa-ff3290d07556" (UID: "0ff521e8-c4d3-40a9-a9fa-ff3290d07556"). InnerVolumeSpecName "kube-api-access-csb5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:04:52.278454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.278432 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util" pod "0ff521e8-c4d3-40a9-a9fa-ff3290d07556" (UID: "0ff521e8-c4d3-40a9-a9fa-ff3290d07556"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:52.374527 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.374497 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:52.374657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.374548 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-util\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:52.374657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:52.374563 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csb5t\" (UniqueName: \"kubernetes.io/projected/0ff521e8-c4d3-40a9-a9fa-ff3290d07556-kube-api-access-csb5t\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:53.043233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.043190 5091 generic.go:356] "Generic (PLEG): container finished" podID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerID="39bc1e2390077d5ea6e6b67580538f0d674037d52cf36b22e57c1b0dfe5bc85a" exitCode=0 Nov 02 15:04:53.043639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.043273 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" event={"ID":"b4b64fc7-406f-48ef-8211-4ecec653e3c7","Type":"ContainerDied","Data":"39bc1e2390077d5ea6e6b67580538f0d674037d52cf36b22e57c1b0dfe5bc85a"} Nov 02 15:04:53.045185 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.045164 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" Nov 02 15:04:53.045283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.045195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7k2tmb" event={"ID":"0ff521e8-c4d3-40a9-a9fa-ff3290d07556","Type":"ContainerDied","Data":"eecda5b92056174d291679009f422f5998cebbced05537587307e45debbdb9d8"} Nov 02 15:04:53.045283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.045230 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eecda5b92056174d291679009f422f5998cebbced05537587307e45debbdb9d8" Nov 02 15:04:53.193831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.193808 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:53.282638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.282609 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle\") pod \"effa60b1-7454-42b9-93b3-cdbf60e6c782\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " Nov 02 15:04:53.282780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.282670 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8\" (UniqueName: \"kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8\") pod \"effa60b1-7454-42b9-93b3-cdbf60e6c782\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " Nov 02 15:04:53.282780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.282710 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util\") pod \"effa60b1-7454-42b9-93b3-cdbf60e6c782\" (UID: \"effa60b1-7454-42b9-93b3-cdbf60e6c782\") " Nov 02 15:04:53.283089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.283063 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle" pod "effa60b1-7454-42b9-93b3-cdbf60e6c782" (UID: "effa60b1-7454-42b9-93b3-cdbf60e6c782"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:53.284702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.284679 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8" pod "effa60b1-7454-42b9-93b3-cdbf60e6c782" (UID: "effa60b1-7454-42b9-93b3-cdbf60e6c782"). InnerVolumeSpecName "kube-api-access-wjdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:04:53.287989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.287967 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util" pod "effa60b1-7454-42b9-93b3-cdbf60e6c782" (UID: "effa60b1-7454-42b9-93b3-cdbf60e6c782"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:53.384196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.384159 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:53.384196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.384191 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjdz8\" (UniqueName: \"kubernetes.io/projected/effa60b1-7454-42b9-93b3-cdbf60e6c782-kube-api-access-wjdz8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:53.384392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:53.384205 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/effa60b1-7454-42b9-93b3-cdbf60e6c782-util\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:54.051301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.051271 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" Nov 02 15:04:54.051301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.051289 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64499q8l" event={"ID":"effa60b1-7454-42b9-93b3-cdbf60e6c782","Type":"ContainerDied","Data":"b957b7a9d864840ba78b2ad5c89817477e3e820dffd7e69050246637f1f09bc1"} Nov 02 15:04:54.051760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.051334 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b957b7a9d864840ba78b2ad5c89817477e3e820dffd7e69050246637f1f09bc1" Nov 02 15:04:54.184985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.184960 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:54.291740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.291669 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle\") pod \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " Nov 02 15:04:54.291740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.291735 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf\" (UniqueName: \"kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf\") pod \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " Nov 02 15:04:54.291933 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.291776 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util\") pod \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\" (UID: \"b4b64fc7-406f-48ef-8211-4ecec653e3c7\") " Nov 02 15:04:54.292331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.292300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle" pod "b4b64fc7-406f-48ef-8211-4ecec653e3c7" (UID: "b4b64fc7-406f-48ef-8211-4ecec653e3c7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:54.293793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.293769 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf" pod "b4b64fc7-406f-48ef-8211-4ecec653e3c7" (UID: "b4b64fc7-406f-48ef-8211-4ecec653e3c7"). InnerVolumeSpecName "kube-api-access-7mrtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:04:54.296919 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.296896 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util" pod "b4b64fc7-406f-48ef-8211-4ecec653e3c7" (UID: "b4b64fc7-406f-48ef-8211-4ecec653e3c7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:04:54.393081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.393039 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-bundle\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:54.393081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.393078 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mrtf\" (UniqueName: \"kubernetes.io/projected/b4b64fc7-406f-48ef-8211-4ecec653e3c7-kube-api-access-7mrtf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:54.393081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:54.393089 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4b64fc7-406f-48ef-8211-4ecec653e3c7-util\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:04:55.057583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:55.057549 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" event={"ID":"b4b64fc7-406f-48ef-8211-4ecec653e3c7","Type":"ContainerDied","Data":"9e011cd7a004be5d664ac100b1a487bb69b90ef8f3d150b3228db8e6d4562b7b"} Nov 02 15:04:55.058028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:55.057604 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e011cd7a004be5d664ac100b1a487bb69b90ef8f3d150b3228db8e6d4562b7b" Nov 02 15:04:55.058028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:04:55.057569 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ch6kf2" Nov 02 15:04:57.936950 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:04:57.936905 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:05:08.585751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.585718 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph"] Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586056 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="util" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586068 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="util" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586079 5091 cpu_manager.go:414] "containerMap: removing container" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="extract" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586084 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="extract" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586095 5091 cpu_manager.go:414] "containerMap: removing container" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="util" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586099 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="util" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586110 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="extract" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586115 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="extract" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586123 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="extract" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586128 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="extract" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586140 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="pull" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586144 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="pull" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586156 5091 cpu_manager.go:414] "containerMap: removing container" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="pull" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586161 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="pull" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586170 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="pull" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586174 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="pull" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586184 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="util" Nov 02 15:05:08.586207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586188 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="util" Nov 02 15:05:08.586747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586237 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="effa60b1-7454-42b9-93b3-cdbf60e6c782" containerName="extract" Nov 02 15:05:08.586747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586245 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff521e8-c4d3-40a9-a9fa-ff3290d07556" containerName="extract" Nov 02 15:05:08.586747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.586250 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b64fc7-406f-48ef-8211-4ecec653e3c7" containerName="extract" Nov 02 15:05:08.589289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.589270 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.691924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.691890 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 15:05:08.692090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.691991 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 15:05:08.692090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.692063 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 15:05:08.692090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.692084 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 15:05:08.692234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.692098 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-bfjvh\"" Nov 02 15:05:08.692234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.692136 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 15:05:08.693675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.693653 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph"] Nov 02 15:05:08.710024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.709998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b33f1072-9373-40f4-ab9b-54f65dba0a19-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.710114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.710049 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33f1072-9373-40f4-ab9b-54f65dba0a19-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.710157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.710115 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t98jm\" (UniqueName: \"kubernetes.io/projected/b33f1072-9373-40f4-ab9b-54f65dba0a19-kube-api-access-t98jm\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.710207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.710171 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.710278 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.710262 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.710315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.710294 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b33f1072-9373-40f4-ab9b-54f65dba0a19-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811222 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33f1072-9373-40f4-ab9b-54f65dba0a19-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811239 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t98jm\" (UniqueName: \"kubernetes.io/projected/b33f1072-9373-40f4-ab9b-54f65dba0a19-kube-api-access-t98jm\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811287 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811308 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.811696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.811660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b33f1072-9373-40f4-ab9b-54f65dba0a19-manifests\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.813494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.813460 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b33f1072-9373-40f4-ab9b-54f65dba0a19-tmp\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.813729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.813710 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.813807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.813791 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-webhook-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.813899 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.813883 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b33f1072-9373-40f4-ab9b-54f65dba0a19-metrics-cert\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.871561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.871535 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t98jm\" (UniqueName: \"kubernetes.io/projected/b33f1072-9373-40f4-ab9b-54f65dba0a19-kube-api-access-t98jm\") pod \"opendatahub-operator-controller-manager-64686958df-cnrph\" (UID: \"b33f1072-9373-40f4-ab9b-54f65dba0a19\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:08.904571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:08.904549 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:09.434379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:09.434347 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph"] Nov 02 15:05:09.439891 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:05:09.439856 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33f1072_9373_40f4_ab9b_54f65dba0a19.slice/crio-9d405bc0d4551d671e8b5e2c18d8c2cd66cd092f376471bd561662f05c97b85c WatchSource:0}: Error finding container 9d405bc0d4551d671e8b5e2c18d8c2cd66cd092f376471bd561662f05c97b85c: Status 404 returned error can't find the container with id 9d405bc0d4551d671e8b5e2c18d8c2cd66cd092f376471bd561662f05c97b85c Nov 02 15:05:10.122718 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:10.122683 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" event={"ID":"b33f1072-9373-40f4-ab9b-54f65dba0a19","Type":"ContainerStarted","Data":"9d405bc0d4551d671e8b5e2c18d8c2cd66cd092f376471bd561662f05c97b85c"} Nov 02 15:05:15.147957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:15.147925 5091 generic.go:356] "Generic (PLEG): container finished" podID="b33f1072-9373-40f4-ab9b-54f65dba0a19" containerID="c99d2fb2d39313637fc4f811511fdfec41a6ae31b221493e896e30cdc5d8c0b4" exitCode=0 Nov 02 15:05:15.148325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:15.147973 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" event={"ID":"b33f1072-9373-40f4-ab9b-54f65dba0a19","Type":"ContainerDied","Data":"c99d2fb2d39313637fc4f811511fdfec41a6ae31b221493e896e30cdc5d8c0b4"} Nov 02 15:05:16.154146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:16.154115 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" event={"ID":"b33f1072-9373-40f4-ab9b-54f65dba0a19","Type":"ContainerStarted","Data":"606b22ab79144d047c563a5c6c67099c35ced831ece504d91b971b6e83d0e5e2"} Nov 02 15:05:16.154597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:16.154321 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:16.392688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:16.392638 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" podStartSLOduration=2.084053965 podStartE2EDuration="8.392624234s" totalImagesPullingTime="5.887380511s" totalInitContainerRuntime="421.189758ms" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:16.344312611 +0000 UTC m=+470.508865148" watchObservedRunningTime="2026-11-02 15:05:16.392624234 +0000 UTC m=+470.557176775" Nov 02 15:05:25.926922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:25.926891 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss"] Nov 02 15:05:25.932931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:25.932911 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:25.983714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:25.983687 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 15:05:26.030697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.030666 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-thddn\"" Nov 02 15:05:26.030846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.030700 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 15:05:26.032491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.032470 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss"] Nov 02 15:05:26.042828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.042808 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Nov 02 15:05:26.052444 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.052421 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5x6\" (UniqueName: \"kubernetes.io/projected/86aa6b27-08ca-42e7-82ad-4347bef0c37d-kube-api-access-zz5x6\") pod \"dns-operator-controller-manager-8557cbdb84-r8wss\" (UID: \"86aa6b27-08ca-42e7-82ad-4347bef0c37d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:26.153012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.152982 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5x6\" (UniqueName: \"kubernetes.io/projected/86aa6b27-08ca-42e7-82ad-4347bef0c37d-kube-api-access-zz5x6\") pod \"dns-operator-controller-manager-8557cbdb84-r8wss\" (UID: \"86aa6b27-08ca-42e7-82ad-4347bef0c37d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:26.219173 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.219098 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5x6\" (UniqueName: \"kubernetes.io/projected/86aa6b27-08ca-42e7-82ad-4347bef0c37d-kube-api-access-zz5x6\") pod \"dns-operator-controller-manager-8557cbdb84-r8wss\" (UID: \"86aa6b27-08ca-42e7-82ad-4347bef0c37d\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:26.245313 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.245282 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:26.619074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:26.619045 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss"] Nov 02 15:05:27.161260 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:27.161229 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-64686958df-cnrph" Nov 02 15:05:27.200448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:27.200415 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" event={"ID":"86aa6b27-08ca-42e7-82ad-4347bef0c37d","Type":"ContainerStarted","Data":"0a38f5115bb5822e860532910c0fee28a5ccc86fcb5c2f61b102cd717f376405"} Nov 02 15:05:27.934170 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:05:27.934132 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:05:30.543341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.543308 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zt62h"] Nov 02 15:05:30.554641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.554617 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:30.594849 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.594820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsls5\" (UniqueName: \"kubernetes.io/projected/07c22d6f-df0c-4478-9683-48ba21cf1493-kube-api-access-dsls5\") pod \"authorino-operator-7b7bb4bc77-zt62h\" (UID: \"07c22d6f-df0c-4478-9683-48ba21cf1493\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:30.651952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.651913 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-22h2d\"" Nov 02 15:05:30.654189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.654162 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zt62h"] Nov 02 15:05:30.696285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.696251 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsls5\" (UniqueName: \"kubernetes.io/projected/07c22d6f-df0c-4478-9683-48ba21cf1493-kube-api-access-dsls5\") pod \"authorino-operator-7b7bb4bc77-zt62h\" (UID: \"07c22d6f-df0c-4478-9683-48ba21cf1493\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:30.761580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.761551 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsls5\" (UniqueName: \"kubernetes.io/projected/07c22d6f-df0c-4478-9683-48ba21cf1493-kube-api-access-dsls5\") pod \"authorino-operator-7b7bb4bc77-zt62h\" (UID: \"07c22d6f-df0c-4478-9683-48ba21cf1493\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:30.870168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:30.869925 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:31.223156 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:31.223068 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" event={"ID":"86aa6b27-08ca-42e7-82ad-4347bef0c37d","Type":"ContainerStarted","Data":"4def463be2d9937ad5623d5673f7e5a1ee5724e4a83417883214535c94ddbbcb"} Nov 02 15:05:31.223317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:31.223203 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:31.253787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:31.253761 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-zt62h"] Nov 02 15:05:31.260231 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:05:31.260208 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c22d6f_df0c_4478_9683_48ba21cf1493.slice/crio-4be7ec4e918ca791fa22baf20abeeec04e6b62adc0b01d21f26845e4ee454210 WatchSource:0}: Error finding container 4be7ec4e918ca791fa22baf20abeeec04e6b62adc0b01d21f26845e4ee454210: Status 404 returned error can't find the container with id 4be7ec4e918ca791fa22baf20abeeec04e6b62adc0b01d21f26845e4ee454210 Nov 02 15:05:31.337366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:31.337322 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" podStartSLOduration=1.972342445 podStartE2EDuration="6.337309099s" totalImagesPullingTime="4.364966654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:31.33703799 +0000 UTC m=+485.501590530" watchObservedRunningTime="2026-11-02 15:05:31.337309099 +0000 UTC m=+485.501861638" Nov 02 15:05:32.229121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:32.229082 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" event={"ID":"07c22d6f-df0c-4478-9683-48ba21cf1493","Type":"ContainerStarted","Data":"4be7ec4e918ca791fa22baf20abeeec04e6b62adc0b01d21f26845e4ee454210"} Nov 02 15:05:35.243577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:35.243507 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" event={"ID":"07c22d6f-df0c-4478-9683-48ba21cf1493","Type":"ContainerStarted","Data":"4e210ae10620bfb939d554de123ed9d0612558fa02bc14df5839394a473e6d7e"} Nov 02 15:05:35.243959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:35.243643 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:35.358043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:35.357992 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" podStartSLOduration=2.186009511 podStartE2EDuration="5.357979141s" totalImagesPullingTime="3.17196963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:35.356195958 +0000 UTC m=+489.520748512" watchObservedRunningTime="2026-11-02 15:05:35.357979141 +0000 UTC m=+489.522531681" Nov 02 15:05:36.269588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.269558 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv"] Nov 02 15:05:36.273004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.272981 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.327399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.327375 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv"] Nov 02 15:05:36.371997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.371971 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jnct4\"" Nov 02 15:05:36.437216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.437187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xr4dv\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.437339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.437218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75hnf\" (UniqueName: \"kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf\") pod \"kuadrant-operator-controller-manager-678d87766f-xr4dv\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.538046 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.537962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xr4dv\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.538046 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.537990 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75hnf\" (UniqueName: \"kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf\") pod \"kuadrant-operator-controller-manager-678d87766f-xr4dv\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.538312 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.538293 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-xr4dv\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.601268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.601238 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75hnf\" (UniqueName: \"kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf\") pod \"kuadrant-operator-controller-manager-678d87766f-xr4dv\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:36.885410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:36.885376 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:37.364138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:37.364059 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv"] Nov 02 15:05:37.370656 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:05:37.370622 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c14a88_be07_4d6f_bef8_7802aa502c48.slice/crio-896754c2d772d145469408f114eae533e7b8e6148f6a70b787e3bd38d834c34a WatchSource:0}: Error finding container 896754c2d772d145469408f114eae533e7b8e6148f6a70b787e3bd38d834c34a: Status 404 returned error can't find the container with id 896754c2d772d145469408f114eae533e7b8e6148f6a70b787e3bd38d834c34a Nov 02 15:05:38.258263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:38.258228 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" event={"ID":"e6c14a88-be07-4d6f-bef8-7802aa502c48","Type":"ContainerStarted","Data":"896754c2d772d145469408f114eae533e7b8e6148f6a70b787e3bd38d834c34a"} Nov 02 15:05:42.232292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:42.232257 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-r8wss" Nov 02 15:05:44.293770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:44.293725 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" event={"ID":"e6c14a88-be07-4d6f-bef8-7802aa502c48","Type":"ContainerStarted","Data":"652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b"} Nov 02 15:05:44.294159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:44.293820 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:44.455884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:44.455830 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" podStartSLOduration=2.567497369 podStartE2EDuration="8.455816619s" totalImagesPullingTime="5.88831925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:44.455695626 +0000 UTC m=+498.620248165" watchObservedRunningTime="2026-11-02 15:05:44.455816619 +0000 UTC m=+498.620369151" Nov 02 15:05:45.217594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.217563 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc"] Nov 02 15:05:45.223005 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.222987 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.277465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.277437 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lx6hj\"" Nov 02 15:05:45.279353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.279323 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc"] Nov 02 15:05:45.310219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310196 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310235 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310258 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310411 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310429 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310510 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310556 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.310781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.310581 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7w7\" (UniqueName: \"kubernetes.io/projected/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-kube-api-access-ng7w7\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411349 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411414 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7w7\" (UniqueName: \"kubernetes.io/projected/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-kube-api-access-ng7w7\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411449 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411558 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411609 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.411919 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411901 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.412031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.411949 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.412095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.412014 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.412145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.412123 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.412276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.412253 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.413713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.413693 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.413933 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.413917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.474389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.474329 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.474732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.474709 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7w7\" (UniqueName: \"kubernetes.io/projected/a43c1886-7c1c-48a9-877e-91d7a4bd6a8d-kube-api-access-ng7w7\") pod \"data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc\" (UID: \"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.535359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.535336 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:45.970565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:45.970512 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc"] Nov 02 15:05:46.010920 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:05:46.010894 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43c1886_7c1c_48a9_877e_91d7a4bd6a8d.slice/crio-4ba2216df31ce19c68f41b94b31d2bcc818d5e7f3af71583ae5e8254ed53acf7 WatchSource:0}: Error finding container 4ba2216df31ce19c68f41b94b31d2bcc818d5e7f3af71583ae5e8254ed53acf7: Status 404 returned error can't find the container with id 4ba2216df31ce19c68f41b94b31d2bcc818d5e7f3af71583ae5e8254ed53acf7 Nov 02 15:05:46.013020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.012988 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:05:46.013092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.013049 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:05:46.013092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.013075 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:05:46.250432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.250345 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-zt62h" Nov 02 15:05:46.305835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.305803 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" event={"ID":"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d","Type":"ContainerStarted","Data":"3cfe7c18da0099922d07af8758e645f315aab4c66a15cd3b94aa61f77ce05289"} Nov 02 15:05:46.306034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.305854 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" event={"ID":"a43c1886-7c1c-48a9-877e-91d7a4bd6a8d","Type":"ContainerStarted","Data":"4ba2216df31ce19c68f41b94b31d2bcc818d5e7f3af71583ae5e8254ed53acf7"} Nov 02 15:05:46.484506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.484458 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" podStartSLOduration=1.484445392 podStartE2EDuration="1.484445392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:46.484333931 +0000 UTC m=+500.648886472" watchObservedRunningTime="2026-11-02 15:05:46.484445392 +0000 UTC m=+500.648997931" Nov 02 15:05:46.535992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:46.535926 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:47.540389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:47.540358 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:48.315384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:48.315356 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:48.316314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:48.316294 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc" Nov 02 15:05:48.971244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:48.971213 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh"] Nov 02 15:05:48.974591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:48.974563 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.024317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.024281 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 02 15:05:49.024317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.024284 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 02 15:05:49.024503 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.024409 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 02 15:05:49.024575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.024511 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-9sdrp\"" Nov 02 15:05:49.041321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.041294 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2fc079a5-1260-497e-94c4-415ce4e43c49-manager-config\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.041467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.041340 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc079a5-1260-497e-94c4-415ce4e43c49-cert\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.041467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.041457 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdfwr\" (UniqueName: \"kubernetes.io/projected/2fc079a5-1260-497e-94c4-415ce4e43c49-kube-api-access-rdfwr\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.041586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.041491 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc079a5-1260-497e-94c4-415ce4e43c49-metrics-certs\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.071683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.071657 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 02 15:05:49.071819 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.071801 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 02 15:05:49.073581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.073557 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh"] Nov 02 15:05:49.142552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.142509 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc079a5-1260-497e-94c4-415ce4e43c49-cert\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.142705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.142618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdfwr\" (UniqueName: \"kubernetes.io/projected/2fc079a5-1260-497e-94c4-415ce4e43c49-kube-api-access-rdfwr\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.142773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.142744 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc079a5-1260-497e-94c4-415ce4e43c49-metrics-certs\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.142830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.142795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2fc079a5-1260-497e-94c4-415ce4e43c49-manager-config\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.143407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.143384 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2fc079a5-1260-497e-94c4-415ce4e43c49-manager-config\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.144978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.144956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2fc079a5-1260-497e-94c4-415ce4e43c49-cert\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.145091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.145062 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc079a5-1260-497e-94c4-415ce4e43c49-metrics-certs\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.204909 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.204878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdfwr\" (UniqueName: \"kubernetes.io/projected/2fc079a5-1260-497e-94c4-415ce4e43c49-kube-api-access-rdfwr\") pod \"jobset-controller-manager-5c66748d65-rclwh\" (UID: \"2fc079a5-1260-497e-94c4-415ce4e43c49\") " pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.287537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.287458 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:49.657930 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:49.657900 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh"] Nov 02 15:05:49.663017 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:05:49.662993 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc079a5_1260_497e_94c4_415ce4e43c49.slice/crio-a7e832939ce19749162ee5396d3500306c976d66992b92512b3d305f6ec64696 WatchSource:0}: Error finding container a7e832939ce19749162ee5396d3500306c976d66992b92512b3d305f6ec64696: Status 404 returned error can't find the container with id a7e832939ce19749162ee5396d3500306c976d66992b92512b3d305f6ec64696 Nov 02 15:05:50.325162 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:50.325119 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" event={"ID":"2fc079a5-1260-497e-94c4-415ce4e43c49","Type":"ContainerStarted","Data":"a7e832939ce19749162ee5396d3500306c976d66992b92512b3d305f6ec64696"} Nov 02 15:05:55.300675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:55.300644 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:05:55.350780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:55.350746 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" event={"ID":"2fc079a5-1260-497e-94c4-415ce4e43c49","Type":"ContainerStarted","Data":"bbd8a3d03a95407813c39f407571229b7761ac466e65134935512f99e2a698e8"} Nov 02 15:05:55.350911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:55.350834 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:05:55.576015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:05:55.575922 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" podStartSLOduration=2.541939803 podStartE2EDuration="7.575908543s" totalImagesPullingTime="5.03396874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:05:55.575777907 +0000 UTC m=+509.740330446" watchObservedRunningTime="2026-11-02 15:05:55.575908543 +0000 UTC m=+509.740461084" Nov 02 15:05:57.933509 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:05:57.933467 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:06:06.361047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:06.361017 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5c66748d65-rclwh" Nov 02 15:06:27.933900 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:06:27.933858 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:06:57.478854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.478822 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r"] Nov 02 15:06:57.482382 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.482359 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.521476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.521449 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.521634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.521484 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.521634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.521549 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgzpw\" (UniqueName: \"kubernetes.io/projected/da5b943e-8fb9-48f3-8757-8a3306654620-kube-api-access-zgzpw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.521710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.521648 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da5b943e-8fb9-48f3-8757-8a3306654620-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.582077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.580941 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 02 15:06:57.582077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.581002 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 02 15:06:57.582077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.581030 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 02 15:06:57.582077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.581064 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 15:06:57.582077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.581500 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 15:06:57.582077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.581922 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-jtr8w\"" Nov 02 15:06:57.588738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.585498 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r"] Nov 02 15:06:57.622261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.622229 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da5b943e-8fb9-48f3-8757-8a3306654620-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.622463 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.622289 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.622463 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.622307 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.622463 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.622343 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgzpw\" (UniqueName: \"kubernetes.io/projected/da5b943e-8fb9-48f3-8757-8a3306654620-kube-api-access-zgzpw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.622463 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:06:57.622380 5091 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 02 15:06:57.622463 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:06:57.622447 5091 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 02 15:06:57.622743 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:06:57.622467 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da5b943e-8fb9-48f3-8757-8a3306654620-tls-cert podName:da5b943e-8fb9-48f3-8757-8a3306654620 nodeName:}" failed. No retries permitted until 2026-11-02 15:06:58.122445939 +0000 UTC m=+572.286998463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/da5b943e-8fb9-48f3-8757-8a3306654620-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" (UID: "da5b943e-8fb9-48f3-8757-8a3306654620") : secret "usage-logs-tenancy-proxy-tls" not found Nov 02 15:06:57.622743 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:06:57.622512 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-proxy-source podName:da5b943e-8fb9-48f3-8757-8a3306654620 nodeName:}" failed. No retries permitted until 2026-11-02 15:06:58.122495411 +0000 UTC m=+572.287047943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" (UID: "da5b943e-8fb9-48f3-8757-8a3306654620") : configmap "usage-tenancy-proxy-source" not found Nov 02 15:06:57.622995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.622976 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.707453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:57.707423 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgzpw\" (UniqueName: \"kubernetes.io/projected/da5b943e-8fb9-48f3-8757-8a3306654620-kube-api-access-zgzpw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:57.933809 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:06:57.933706 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:06:58.127856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:58.127817 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da5b943e-8fb9-48f3-8757-8a3306654620-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:58.128055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:58.127883 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:58.128687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:58.128660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/da5b943e-8fb9-48f3-8757-8a3306654620-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:58.130265 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:58.130244 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/da5b943e-8fb9-48f3-8757-8a3306654620-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-xqw9r\" (UID: \"da5b943e-8fb9-48f3-8757-8a3306654620\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:58.394733 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:58.394698 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:06:58.765387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:58.765311 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r"] Nov 02 15:06:58.770946 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:06:58.770917 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5b943e_8fb9_48f3_8757_8a3306654620.slice/crio-ed4162435c4204dd4dbbf4bcd0e53e836c620cde2339fe0eab368d5cda9907a2 WatchSource:0}: Error finding container ed4162435c4204dd4dbbf4bcd0e53e836c620cde2339fe0eab368d5cda9907a2: Status 404 returned error can't find the container with id ed4162435c4204dd4dbbf4bcd0e53e836c620cde2339fe0eab368d5cda9907a2 Nov 02 15:06:59.651951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:06:59.651915 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" event={"ID":"da5b943e-8fb9-48f3-8757-8a3306654620","Type":"ContainerStarted","Data":"ed4162435c4204dd4dbbf4bcd0e53e836c620cde2339fe0eab368d5cda9907a2"} Nov 02 15:07:02.660218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.660186 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd"] Nov 02 15:07:02.665096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.665071 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.763125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.763085 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 02 15:07:02.763281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.763136 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kxk6h\"" Nov 02 15:07:02.765129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.765102 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd"] Nov 02 15:07:02.765613 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.765590 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9hzkd\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.765708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.765618 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9bx\" (UniqueName: \"kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx\") pod \"llmisvc-controller-manager-869d5dcc5d-9hzkd\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.867171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.867128 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9hzkd\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.867171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.867173 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9bx\" (UniqueName: \"kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx\") pod \"llmisvc-controller-manager-869d5dcc5d-9hzkd\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.869851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.869826 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-9hzkd\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.926246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.926181 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9bx\" (UniqueName: \"kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx\") pod \"llmisvc-controller-manager-869d5dcc5d-9hzkd\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:02.978167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:02.978135 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:03.362208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:03.362123 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd"] Nov 02 15:07:03.369897 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:07:03.369861 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod09e46849_a94f_4ca0_add9_126da6039f69.slice/crio-d61b684d2adf91c2d5ac6f5aca52d5d5332aaa23bb661da4d000f36aa0b5375b WatchSource:0}: Error finding container d61b684d2adf91c2d5ac6f5aca52d5d5332aaa23bb661da4d000f36aa0b5375b: Status 404 returned error can't find the container with id d61b684d2adf91c2d5ac6f5aca52d5d5332aaa23bb661da4d000f36aa0b5375b Nov 02 15:07:03.677402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:03.677315 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" event={"ID":"09e46849-a94f-4ca0-add9-126da6039f69","Type":"ContainerStarted","Data":"d61b684d2adf91c2d5ac6f5aca52d5d5332aaa23bb661da4d000f36aa0b5375b"} Nov 02 15:07:07.710798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:07.710759 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" event={"ID":"09e46849-a94f-4ca0-add9-126da6039f69","Type":"ContainerStarted","Data":"dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4"} Nov 02 15:07:07.711228 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:07.710908 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:07.822612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:07.822489 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" podStartSLOduration=1.709969184 podStartE2EDuration="5.822474103s" totalImagesPullingTime="4.112504919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:07:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:07.82166914 +0000 UTC m=+581.986221681" watchObservedRunningTime="2026-11-02 15:07:07.822474103 +0000 UTC m=+581.987026645" Nov 02 15:07:12.745206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:12.745170 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" event={"ID":"da5b943e-8fb9-48f3-8757-8a3306654620","Type":"ContainerStarted","Data":"6d9e16746f7b429f311cd7a190f880552bd98c77c2c4355867fb070b29751a13"} Nov 02 15:07:12.745604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:12.745267 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:07:12.876801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:12.876757 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" podStartSLOduration=2.041943365 podStartE2EDuration="15.876743614s" totalImagesPullingTime="13.834800249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:07:12.875645907 +0000 UTC m=+587.040198446" watchObservedRunningTime="2026-11-02 15:07:12.876743614 +0000 UTC m=+587.041296153" Nov 02 15:07:23.755800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:23.755763 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-xqw9r" Nov 02 15:07:26.004140 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:26.004105 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:07:26.005460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:26.005436 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:07:27.933317 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:07:27.933276 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:07:38.719601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:38.719558 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:07:45.906344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:45.906312 5091 generic.go:356] "Generic (PLEG): container finished" podID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerID="3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883" exitCode=0 Nov 02 15:07:45.906845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:45.906350 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" event={"ID":"17a5dc38-af8a-4235-9d70-ed61bb284a19","Type":"ContainerDied","Data":"3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883"} Nov 02 15:07:45.906845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:45.906679 5091 scope.go:122] "RemoveContainer" containerID="3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883" Nov 02 15:07:45.907736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:45.907709 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:07:45.907815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:45.907771 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:07:45.907815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:45.907800 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:07:46.913328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:46.913292 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" event={"ID":"17a5dc38-af8a-4235-9d70-ed61bb284a19","Type":"ContainerStarted","Data":"d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca"} Nov 02 15:07:47.049538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:47.044961 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:07:47.049538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:47.045099 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k7pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:47.049538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:47.045140 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 15:07:48.050542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:48.049936 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k7pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:48.050542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:48.050005 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 15:07:49.046540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:49.044651 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k7pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:49.046540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:49.044728 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 15:07:50.044930 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:50.044892 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k7pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:50.045551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:50.045499 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 15:07:51.051924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:51.051883 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-k7pr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 02 15:07:51.052333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:51.051965 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 02 15:07:52.048482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:52.048450 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:07:52.048907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:52.048882 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:07:52.049664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:07:52.049606 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:07:57.933621 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:07:57.933581 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:08:26.189662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.189624 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-748vc"] Nov 02 15:08:26.193291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.193275 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.226257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.226230 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndrn\" (UniqueName: \"kubernetes.io/projected/0243dab1-1945-4a78-9ef4-d04c99c2cb26-kube-api-access-hndrn\") pod \"llmisvc-controller-manager-6466c554d8-748vc\" (UID: \"0243dab1-1945-4a78-9ef4-d04c99c2cb26\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.226381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.226290 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0243dab1-1945-4a78-9ef4-d04c99c2cb26-cert\") pod \"llmisvc-controller-manager-6466c554d8-748vc\" (UID: \"0243dab1-1945-4a78-9ef4-d04c99c2cb26\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.248362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.248333 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-748vc"] Nov 02 15:08:26.326863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.326828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndrn\" (UniqueName: \"kubernetes.io/projected/0243dab1-1945-4a78-9ef4-d04c99c2cb26-kube-api-access-hndrn\") pod \"llmisvc-controller-manager-6466c554d8-748vc\" (UID: \"0243dab1-1945-4a78-9ef4-d04c99c2cb26\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.327042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.326886 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0243dab1-1945-4a78-9ef4-d04c99c2cb26-cert\") pod \"llmisvc-controller-manager-6466c554d8-748vc\" (UID: \"0243dab1-1945-4a78-9ef4-d04c99c2cb26\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.329332 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.329312 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0243dab1-1945-4a78-9ef4-d04c99c2cb26-cert\") pod \"llmisvc-controller-manager-6466c554d8-748vc\" (UID: \"0243dab1-1945-4a78-9ef4-d04c99c2cb26\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.385360 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.385325 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndrn\" (UniqueName: \"kubernetes.io/projected/0243dab1-1945-4a78-9ef4-d04c99c2cb26-kube-api-access-hndrn\") pod \"llmisvc-controller-manager-6466c554d8-748vc\" (UID: \"0243dab1-1945-4a78-9ef4-d04c99c2cb26\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.505644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.505558 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:26.877310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:26.877278 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-748vc"] Nov 02 15:08:26.887082 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:08:26.887053 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0243dab1_1945_4a78_9ef4_d04c99c2cb26.slice/crio-e2caef7e03f75771b9f4cb078e9e4a3718c24d91173ca7b9d9fd42d650a76168 WatchSource:0}: Error finding container e2caef7e03f75771b9f4cb078e9e4a3718c24d91173ca7b9d9fd42d650a76168: Status 404 returned error can't find the container with id e2caef7e03f75771b9f4cb078e9e4a3718c24d91173ca7b9d9fd42d650a76168 Nov 02 15:08:27.125618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:27.125582 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" event={"ID":"0243dab1-1945-4a78-9ef4-d04c99c2cb26","Type":"ContainerStarted","Data":"e2caef7e03f75771b9f4cb078e9e4a3718c24d91173ca7b9d9fd42d650a76168"} Nov 02 15:08:27.933507 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:08:27.933472 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:08:28.131418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:28.131381 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" event={"ID":"0243dab1-1945-4a78-9ef4-d04c99c2cb26","Type":"ContainerStarted","Data":"461666eaf898948ca7299fac0d226487aee680eb726629126e649de5eb88c0c5"} Nov 02 15:08:28.131649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:28.131466 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:28.243268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:28.243222 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" podStartSLOduration=1.397740218 podStartE2EDuration="2.24320997s" totalImagesPullingTime="845.469752ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:08:28.242341538 +0000 UTC m=+662.406894078" watchObservedRunningTime="2026-11-02 15:08:28.24320997 +0000 UTC m=+662.407762510" Nov 02 15:08:57.933947 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:08:57.933862 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:08:59.139526 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.139486 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-748vc" Nov 02 15:08:59.300944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.300908 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd"] Nov 02 15:08:59.301193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.301146 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" podUID="09e46849-a94f-4ca0-add9-126da6039f69" containerName="manager" containerID="cri-o://dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4" gracePeriod=30 Nov 02 15:08:59.546378 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.546356 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:08:59.600979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.600953 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert\" (UniqueName: \"kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert\") pod \"09e46849-a94f-4ca0-add9-126da6039f69\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " Nov 02 15:08:59.601115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.600991 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx\" (UniqueName: \"kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx\") pod \"09e46849-a94f-4ca0-add9-126da6039f69\" (UID: \"09e46849-a94f-4ca0-add9-126da6039f69\") " Nov 02 15:08:59.603092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.603067 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert" pod "09e46849-a94f-4ca0-add9-126da6039f69" (UID: "09e46849-a94f-4ca0-add9-126da6039f69"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:08:59.603178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.603147 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx" pod "09e46849-a94f-4ca0-add9-126da6039f69" (UID: "09e46849-a94f-4ca0-add9-126da6039f69"). InnerVolumeSpecName "kube-api-access-kp9bx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:08:59.702249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.702183 5091 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09e46849-a94f-4ca0-add9-126da6039f69-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:08:59.702249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:08:59.702212 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9bx\" (UniqueName: \"kubernetes.io/projected/09e46849-a94f-4ca0-add9-126da6039f69-kube-api-access-kp9bx\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:09:00.300902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.300809 5091 generic.go:356] "Generic (PLEG): container finished" podID="09e46849-a94f-4ca0-add9-126da6039f69" containerID="dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4" exitCode=0 Nov 02 15:09:00.300902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.300864 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" Nov 02 15:09:00.300902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.300890 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" event={"ID":"09e46849-a94f-4ca0-add9-126da6039f69","Type":"ContainerDied","Data":"dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4"} Nov 02 15:09:00.301452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.300959 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd" event={"ID":"09e46849-a94f-4ca0-add9-126da6039f69","Type":"ContainerDied","Data":"d61b684d2adf91c2d5ac6f5aca52d5d5332aaa23bb661da4d000f36aa0b5375b"} Nov 02 15:09:00.301452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.300979 5091 scope.go:122] "RemoveContainer" containerID="dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4" Nov 02 15:09:00.309538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.309495 5091 scope.go:122] "RemoveContainer" containerID="dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4" Nov 02 15:09:00.309924 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:09:00.309889 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4\": container with ID starting with dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4 not found: ID does not exist" containerID="dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4" Nov 02 15:09:00.309981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.309952 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4"} err="failed to get container status \"dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4\": rpc error: code = NotFound desc = could not find container \"dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4\": container with ID starting with dd75039052beb3a308eb2a544b37559ac2b08f0902c25f776c4222f69ab0e6d4 not found: ID does not exist" Nov 02 15:09:00.457034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.457001 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd"] Nov 02 15:09:00.504120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:00.504090 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-9hzkd"] Nov 02 15:09:01.958228 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:09:01.958189 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e46849-a94f-4ca0-add9-126da6039f69" path="/var/lib/kubelet/pods/09e46849-a94f-4ca0-add9-126da6039f69/volumes" Nov 02 15:09:27.936062 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:09:27.936019 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:09:57.934055 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:09:57.934003 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:10:27.933859 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:10:27.933819 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:10:57.933192 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:10:57.933143 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:11:27.935225 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:11:27.935185 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:11:43.873536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.873449 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72"] Nov 02 15:11:43.873900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.873878 5091 cpu_manager.go:414] "containerMap: removing container" podUID="09e46849-a94f-4ca0-add9-126da6039f69" containerName="manager" Nov 02 15:11:43.873900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.873895 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e46849-a94f-4ca0-add9-126da6039f69" containerName="manager" Nov 02 15:11:43.874048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.874035 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e46849-a94f-4ca0-add9-126da6039f69" containerName="manager" Nov 02 15:11:43.877098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.877078 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:43.975943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.975914 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-891723\"" Nov 02 15:11:43.976097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.976036 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-891723-dockercfg-b7z2n\"" Nov 02 15:11:43.978486 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:43.978463 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72"] Nov 02 15:11:44.005414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.005390 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qmq\" (UniqueName: \"kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq\") pod \"payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.005563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.005434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.025026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.025002 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz"] Nov 02 15:11:44.028477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.028460 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.085693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.085659 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz"] Nov 02 15:11:44.106006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.105978 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qmq\" (UniqueName: \"kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq\") pod \"payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.106125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.106024 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.106125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.106077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdt5x\" (UniqueName: \"kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x\") pod \"payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.106125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.106101 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.106744 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.106726 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.165417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.165364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qmq\" (UniqueName: \"kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq\") pod \"payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.206287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.206267 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:44.206527 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.206494 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.206671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.206652 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdt5x\" (UniqueName: \"kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x\") pod \"payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.207306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.207284 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.265188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.265161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdt5x\" (UniqueName: \"kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x\") pod \"payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.341951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.341919 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:44.578474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.578398 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72"] Nov 02 15:11:44.583360 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:11:44.583335 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08a7a22_5b7e_48b9_9fa2_33ea61732157.slice/crio-fc0c8880e6d7f41f83fdc1c8e68d58ba2e4c21b3875acdde70341c805e946a72 WatchSource:0}: Error finding container fc0c8880e6d7f41f83fdc1c8e68d58ba2e4c21b3875acdde70341c805e946a72: Status 404 returned error can't find the container with id fc0c8880e6d7f41f83fdc1c8e68d58ba2e4c21b3875acdde70341c805e946a72 Nov 02 15:11:44.585227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.585212 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:11:44.711757 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:44.711724 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz"] Nov 02 15:11:44.715853 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:11:44.715829 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96bc8d73_120a_4019_a9fd_41d3265d7a39.slice/crio-d4632fc5c4626185d09015284b43a743e2720e9e4a0b676075b4362032929fe4 WatchSource:0}: Error finding container d4632fc5c4626185d09015284b43a743e2720e9e4a0b676075b4362032929fe4: Status 404 returned error can't find the container with id d4632fc5c4626185d09015284b43a743e2720e9e4a0b676075b4362032929fe4 Nov 02 15:11:45.117122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:45.117088 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" event={"ID":"a08a7a22-5b7e-48b9-9fa2-33ea61732157","Type":"ContainerStarted","Data":"fc0c8880e6d7f41f83fdc1c8e68d58ba2e4c21b3875acdde70341c805e946a72"} Nov 02 15:11:45.118251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:45.118222 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" event={"ID":"96bc8d73-120a-4019-a9fd-41d3265d7a39","Type":"ContainerStarted","Data":"d4632fc5c4626185d09015284b43a743e2720e9e4a0b676075b4362032929fe4"} Nov 02 15:11:48.139298 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.139250 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" event={"ID":"96bc8d73-120a-4019-a9fd-41d3265d7a39","Type":"ContainerStarted","Data":"785fbe68559036c4601f16d303c13d38091ecb6dd547959de1ae9c3f0d28d0d9"} Nov 02 15:11:48.139798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.139355 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:48.141022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.140994 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" event={"ID":"a08a7a22-5b7e-48b9-9fa2-33ea61732157","Type":"ContainerStarted","Data":"f119362dd89f91f8ccfa8921b2659550d267c9b82ec625494a6814124d42f843"} Nov 02 15:11:48.141156 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.141137 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:11:48.264480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.264426 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" podStartSLOduration=2.304170576 podStartE2EDuration="5.264413564s" totalImagesPullingTime="2.960242988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:48.260830154 +0000 UTC m=+862.425382694" watchObservedRunningTime="2026-11-02 15:11:48.264413564 +0000 UTC m=+862.428966103" Nov 02 15:11:48.376548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.376460 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" podStartSLOduration=2.409623324 podStartE2EDuration="5.376426762s" totalImagesPullingTime="2.966803438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:48.372102181 +0000 UTC m=+862.536654718" watchObservedRunningTime="2026-11-02 15:11:48.376426762 +0000 UTC m=+862.540979303" Nov 02 15:11:48.629355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.629308 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d"] Nov 02 15:11:48.633181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.633162 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.687947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.687910 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-624908-dockercfg-29f4c\"" Nov 02 15:11:48.688780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.688075 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-624908\"" Nov 02 15:11:48.689877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.689847 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d"] Nov 02 15:11:48.749990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.749949 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.750175 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.750116 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8sml\" (UniqueName: \"kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml\") pod \"payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.774825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.774787 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8"] Nov 02 15:11:48.778956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.778930 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:48.833974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.833748 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8"] Nov 02 15:11:48.850871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.850839 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kz82\" (UniqueName: \"kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82\") pod \"payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:48.850987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.850969 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8sml\" (UniqueName: \"kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml\") pod \"payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.851095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.851074 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.851164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.851111 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:48.851952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.851931 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.910331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.910275 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8sml\" (UniqueName: \"kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml\") pod \"payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.946196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.946166 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:48.952705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.952680 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:48.952772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.952727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kz82\" (UniqueName: \"kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82\") pod \"payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:48.953319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:48.953302 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:49.015407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:49.015375 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kz82\" (UniqueName: \"kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82\") pod \"payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:49.092283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:49.092253 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:49.315023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:49.314952 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d"] Nov 02 15:11:49.321481 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:11:49.321457 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4746e31d_ebfd_4b9d_af84_9449d4690554.slice/crio-bee657a8f25331e2a050c07347b0024cf0829d7fee198c166799e961712e413b WatchSource:0}: Error finding container bee657a8f25331e2a050c07347b0024cf0829d7fee198c166799e961712e413b: Status 404 returned error can't find the container with id bee657a8f25331e2a050c07347b0024cf0829d7fee198c166799e961712e413b Nov 02 15:11:49.464744 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:49.464717 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8"] Nov 02 15:11:49.471645 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:11:49.471613 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf255e40_df8e_45d8_9eae_9c9bdcb78a29.slice/crio-1795478c70533345187ff8d893bb43dbe10f42123e5a42fb9c39240f49b8eeee WatchSource:0}: Error finding container 1795478c70533345187ff8d893bb43dbe10f42123e5a42fb9c39240f49b8eeee: Status 404 returned error can't find the container with id 1795478c70533345187ff8d893bb43dbe10f42123e5a42fb9c39240f49b8eeee Nov 02 15:11:50.155165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.155130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" event={"ID":"bf255e40-df8e-45d8-9eae-9c9bdcb78a29","Type":"ContainerStarted","Data":"32de910863a57acbea0200bfa29555cd5969faa308ce41218872fa9a404fece9"} Nov 02 15:11:50.155370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.155192 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" event={"ID":"bf255e40-df8e-45d8-9eae-9c9bdcb78a29","Type":"ContainerStarted","Data":"1795478c70533345187ff8d893bb43dbe10f42123e5a42fb9c39240f49b8eeee"} Nov 02 15:11:50.155370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.155235 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:11:50.156626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.156599 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" event={"ID":"4746e31d-ebfd-4b9d-af84-9449d4690554","Type":"ContainerStarted","Data":"6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4"} Nov 02 15:11:50.156722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.156645 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" event={"ID":"4746e31d-ebfd-4b9d-af84-9449d4690554","Type":"ContainerStarted","Data":"bee657a8f25331e2a050c07347b0024cf0829d7fee198c166799e961712e413b"} Nov 02 15:11:50.156722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.156674 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:11:50.269548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.269473 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" podStartSLOduration=2.269454447 podStartE2EDuration="2.269454447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:50.268204337 +0000 UTC m=+864.432756878" watchObservedRunningTime="2026-11-02 15:11:50.269454447 +0000 UTC m=+864.434006988" Nov 02 15:11:50.379878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:50.379825 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" podStartSLOduration=2.379809281 podStartE2EDuration="2.379809281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:50.379369333 +0000 UTC m=+864.543921873" watchObservedRunningTime="2026-11-02 15:11:50.379809281 +0000 UTC m=+864.544361833" Nov 02 15:11:51.075109 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.075076 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw"] Nov 02 15:11:51.078737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.078718 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.129610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.129584 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-71d1eb-dockercfg-rr649\"" Nov 02 15:11:51.171354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.171325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.171611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.171477 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcx55\" (UniqueName: \"kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55\") pod \"payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.176861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.176833 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-71d1eb\"" Nov 02 15:11:51.178954 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.178931 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw"] Nov 02 15:11:51.236351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.236319 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6"] Nov 02 15:11:51.240068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.240049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.273056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.273016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcx55\" (UniqueName: \"kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55\") pod \"payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.273226 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.273074 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.273829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.273805 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.295973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.295926 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6"] Nov 02 15:11:51.331929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.331867 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcx55\" (UniqueName: \"kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55\") pod \"payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.373576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.373547 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvxkm\" (UniqueName: \"kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm\") pod \"payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.373702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.373686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.392672 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.392648 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:51.474715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.474684 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvxkm\" (UniqueName: \"kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm\") pod \"payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.474990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.474971 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.475805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.475776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.534442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.534408 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvxkm\" (UniqueName: \"kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm\") pod \"payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.553398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.553372 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:51.765351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.765315 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw"] Nov 02 15:11:51.771291 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:11:51.771148 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode912adb2_3cbb_4346_a85f_f05e46536ae8.slice/crio-8019e3de611a5927c4519688ec7430a225fe5870b73234d6c9cba15a3a53d20b WatchSource:0}: Error finding container 8019e3de611a5927c4519688ec7430a225fe5870b73234d6c9cba15a3a53d20b: Status 404 returned error can't find the container with id 8019e3de611a5927c4519688ec7430a225fe5870b73234d6c9cba15a3a53d20b Nov 02 15:11:51.932224 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:51.932193 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6"] Nov 02 15:11:51.938036 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:11:51.938005 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c95b5_c394_4866_af0b_b5cc338d0543.slice/crio-52ba93d26f968b7dfc1de7ac4196485d0b254a3f67e7ee7d580b1dd24b20887b WatchSource:0}: Error finding container 52ba93d26f968b7dfc1de7ac4196485d0b254a3f67e7ee7d580b1dd24b20887b: Status 404 returned error can't find the container with id 52ba93d26f968b7dfc1de7ac4196485d0b254a3f67e7ee7d580b1dd24b20887b Nov 02 15:11:52.171178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.171136 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" event={"ID":"379c95b5-c394-4866-af0b-b5cc338d0543","Type":"ContainerStarted","Data":"963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968"} Nov 02 15:11:52.171387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.171213 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:11:52.171387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.171240 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" event={"ID":"379c95b5-c394-4866-af0b-b5cc338d0543","Type":"ContainerStarted","Data":"52ba93d26f968b7dfc1de7ac4196485d0b254a3f67e7ee7d580b1dd24b20887b"} Nov 02 15:11:52.172907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.172880 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" event={"ID":"e912adb2-3cbb-4346-a85f-f05e46536ae8","Type":"ContainerStarted","Data":"a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6"} Nov 02 15:11:52.173018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.172928 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" event={"ID":"e912adb2-3cbb-4346-a85f-f05e46536ae8","Type":"ContainerStarted","Data":"8019e3de611a5927c4519688ec7430a225fe5870b73234d6c9cba15a3a53d20b"} Nov 02 15:11:52.173018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.172967 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:11:52.285072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.284969 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" podStartSLOduration=1.284954469 podStartE2EDuration="1.284954469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:52.282337853 +0000 UTC m=+866.446890393" watchObservedRunningTime="2026-11-02 15:11:52.284954469 +0000 UTC m=+866.449507009" Nov 02 15:11:52.395450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:52.395342 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" podStartSLOduration=1.395326805 podStartE2EDuration="1.395326805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:11:52.394931325 +0000 UTC m=+866.559483868" watchObservedRunningTime="2026-11-02 15:11:52.395326805 +0000 UTC m=+866.559879373" Nov 02 15:11:57.933910 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:11:57.933870 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:11:59.152229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:59.152200 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:11:59.152796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:11:59.152268 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:12:01.169061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:01.169029 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:12:01.169540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:01.169484 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:12:03.181976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:03.181945 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:12:03.182380 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:03.182348 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:12:17.807685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.807641 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq"] Nov 02 15:12:17.811199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.811177 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:17.903690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.903658 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:17.903856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.903696 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:17.903856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.903715 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blrrt\" (UniqueName: \"kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:17.903943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.903855 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:17.909722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.909695 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:12:17.909840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.909721 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-624908-serving-cert\"" Nov 02 15:12:17.909840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.909726 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:12:17.909915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.909908 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wj8b6\"" Nov 02 15:12:17.910104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.910088 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 15:12:17.912310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:17.912294 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq"] Nov 02 15:12:18.005193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.005160 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.005390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.005207 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.005390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.005330 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blrrt\" (UniqueName: \"kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.005509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.005428 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.005609 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:12:18.005552 5091 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-serving-cert: secret "maas-api-e2e-worker-w1-624908-serving-cert" not found Nov 02 15:12:18.005664 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:12:18.005642 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls podName:f65c0e09-68e3-4f91-847c-5175c4f18e78 nodeName:}" failed. No retries permitted until 2026-11-02 15:12:18.505614793 +0000 UTC m=+892.670167324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls") pod "maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" (UID: "f65c0e09-68e3-4f91-847c-5175c4f18e78") : secret "maas-api-e2e-worker-w1-624908-serving-cert" not found Nov 02 15:12:18.005946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.005925 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.007722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.007700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.064268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.064203 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blrrt\" (UniqueName: \"kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.510021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.509987 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.512529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.512496 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\") pod \"maas-api-e2e-worker-w1-624908-78c5986487-k9sfq\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:18.724445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:18.724414 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:19.099353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:19.099274 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq"] Nov 02 15:12:19.104587 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:12:19.104559 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65c0e09_68e3_4f91_847c_5175c4f18e78.slice/crio-656fa444a33a86f5729309034de36e4900becd6c08360e1a4d0fc55bc918763d WatchSource:0}: Error finding container 656fa444a33a86f5729309034de36e4900becd6c08360e1a4d0fc55bc918763d: Status 404 returned error can't find the container with id 656fa444a33a86f5729309034de36e4900becd6c08360e1a4d0fc55bc918763d Nov 02 15:12:19.313493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:19.313453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" event={"ID":"f65c0e09-68e3-4f91-847c-5175c4f18e78","Type":"ContainerStarted","Data":"656fa444a33a86f5729309034de36e4900becd6c08360e1a4d0fc55bc918763d"} Nov 02 15:12:22.333888 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:22.333849 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" event={"ID":"f65c0e09-68e3-4f91-847c-5175c4f18e78","Type":"ContainerStarted","Data":"93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f"} Nov 02 15:12:22.334279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:22.333917 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:22.449067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:22.448979 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" podStartSLOduration=2.4346028840000002 podStartE2EDuration="5.448965037s" totalImagesPullingTime="3.014362153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:22.447703602 +0000 UTC m=+896.612256141" watchObservedRunningTime="2026-11-02 15:12:22.448965037 +0000 UTC m=+896.613517580" Nov 02 15:12:26.046030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:26.045998 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:12:26.048212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:26.048185 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:12:27.937824 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:12:27.937783 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:12:28.345761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.345688 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:12:28.445104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.445072 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6"] Nov 02 15:12:28.449779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.449759 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.494915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.494870 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.494915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.494917 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.495129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.494998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttnj\" (UniqueName: \"kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.495129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.495029 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.495129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.495058 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.495270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.495176 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.508730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.508698 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6"] Nov 02 15:12:28.511012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.510989 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"kube-root-ca.crt\"" Nov 02 15:12:28.511191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.511000 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"openshift-service-ca.crt\"" Nov 02 15:12:28.519044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.519019 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"default-dockercfg-2ssb5\"" Nov 02 15:12:28.519165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.519134 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"simulator-w4-8a2e6f-kserve-self-signed-certs\"" Nov 02 15:12:28.596943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.596859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.596943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.596915 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.596976 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.597089 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.597106 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.597125 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttnj\" (UniqueName: \"kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.597672 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.597679 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.597960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.597910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.599665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.599639 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.599774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.599724 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.660090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.660064 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttnj\" (UniqueName: \"kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj\") pod \"simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:28.780162 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:28.780128 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:29.157702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:29.157657 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6"] Nov 02 15:12:29.162338 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:12:29.162312 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf795812_c210_47d8_8a8e_d4aa5546a4a8.slice/crio-fc991b34a0b51b23991966160abdfe665a36108f82102b2cd67b2a3c0e9ec8bb WatchSource:0}: Error finding container fc991b34a0b51b23991966160abdfe665a36108f82102b2cd67b2a3c0e9ec8bb: Status 404 returned error can't find the container with id fc991b34a0b51b23991966160abdfe665a36108f82102b2cd67b2a3c0e9ec8bb Nov 02 15:12:29.371831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:29.371786 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" event={"ID":"bf795812-c210-47d8-8a8e-d4aa5546a4a8","Type":"ContainerStarted","Data":"fc991b34a0b51b23991966160abdfe665a36108f82102b2cd67b2a3c0e9ec8bb"} Nov 02 15:12:31.386328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:31.386293 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" event={"ID":"bf795812-c210-47d8-8a8e-d4aa5546a4a8","Type":"ContainerStarted","Data":"f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263"} Nov 02 15:12:31.498218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:31.498164 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podStartSLOduration=1.9271957720000001 podStartE2EDuration="3.498149511s" totalImagesPullingTime="1.570953739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:31.497158865 +0000 UTC m=+905.661711405" watchObservedRunningTime="2026-11-02 15:12:31.498149511 +0000 UTC m=+905.662702051" Nov 02 15:12:31.780836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:31.780756 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:38.780871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:38.780831 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:38.793851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:38.793827 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:39.451544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:39.451501 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:12:55.025104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.025067 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27"] Nov 02 15:12:55.030991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.030972 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.054421 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.054392 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.054586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.054430 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.054755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.054724 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdgc\" (UniqueName: \"kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.054895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.054858 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.054895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.054887 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.054989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.054909 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.129314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.129280 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"simulator-w2-fce776-kserve-self-signed-certs\"" Nov 02 15:12:55.129482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.129362 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"default-dockercfg-fqrg8\"" Nov 02 15:12:55.129482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.129375 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"kube-root-ca.crt\"" Nov 02 15:12:55.129482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.129364 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"openshift-service-ca.crt\"" Nov 02 15:12:55.131444 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.131426 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27"] Nov 02 15:12:55.156418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156392 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.156582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156427 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.156582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.156582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156511 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.156828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156659 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdgc\" (UniqueName: \"kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.156828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156755 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.156960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.156940 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.157022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.157001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.157107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.157091 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.158727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.158701 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.159215 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.159195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.215195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.215161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdgc\" (UniqueName: \"kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc\") pod \"simulator-w2-fce776-kserve-79b95b659-x6s27\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.343211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.343124 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:55.717185 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:55.717149 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27"] Nov 02 15:12:55.723425 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:12:55.723392 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b9b569c_2c19_48e7_bb05_109719649670.slice/crio-faf4995c3bd85cedcf717a522b8f38ed68c09e7e91a25a1ca16ba135ad0c07f1 WatchSource:0}: Error finding container faf4995c3bd85cedcf717a522b8f38ed68c09e7e91a25a1ca16ba135ad0c07f1: Status 404 returned error can't find the container with id faf4995c3bd85cedcf717a522b8f38ed68c09e7e91a25a1ca16ba135ad0c07f1 Nov 02 15:12:56.529961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:56.529924 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" event={"ID":"6b9b569c-2c19-48e7-bb05-109719649670","Type":"ContainerStarted","Data":"e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722"} Nov 02 15:12:56.529961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:56.529966 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" event={"ID":"6b9b569c-2c19-48e7-bb05-109719649670","Type":"ContainerStarted","Data":"faf4995c3bd85cedcf717a522b8f38ed68c09e7e91a25a1ca16ba135ad0c07f1"} Nov 02 15:12:56.645169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:56.645109 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podStartSLOduration=2.645095107 podStartE2EDuration="2.645095107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:12:56.643817311 +0000 UTC m=+930.808369853" watchObservedRunningTime="2026-11-02 15:12:56.645095107 +0000 UTC m=+930.809647641" Nov 02 15:12:57.343243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:12:57.343211 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:12:57.934924 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:12:57.934883 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:13:02.230899 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.230864 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq"] Nov 02 15:13:02.235097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.235072 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.327379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.327314 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.327379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.327357 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.327725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.327504 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.327725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.327563 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.327725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.327594 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.327725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.327658 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5xrm\" (UniqueName: \"kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.334866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.334844 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"default-dockercfg-z5s42\"" Nov 02 15:13:02.334975 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.334867 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"kube-root-ca.crt\"" Nov 02 15:13:02.335038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.334983 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"openshift-service-ca.crt\"" Nov 02 15:13:02.335038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.334986 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-71d1eb\"/\"simulator-w3-71d1eb-kserve-self-signed-certs\"" Nov 02 15:13:02.337329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.337307 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq"] Nov 02 15:13:02.429047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429010 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429055 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429112 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429131 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5xrm\" (UniqueName: \"kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429447 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429498 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.429641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.429611 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.431600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.431575 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.431894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.431875 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.488659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.488578 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5xrm\" (UniqueName: \"kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm\") pod \"simulator-w3-71d1eb-kserve-5457877df4-582mq\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.547740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.547709 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:02.937635 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:02.937600 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq"] Nov 02 15:13:02.943875 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:13:02.943849 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d0d0b6_8ac6_4c8a_8784_d509c9838c18.slice/crio-77c32135c4a1c7937e5110a942e9ef3c2b388a0a0e5390d7cb8ed73e9bbd710d WatchSource:0}: Error finding container 77c32135c4a1c7937e5110a942e9ef3c2b388a0a0e5390d7cb8ed73e9bbd710d: Status 404 returned error can't find the container with id 77c32135c4a1c7937e5110a942e9ef3c2b388a0a0e5390d7cb8ed73e9bbd710d Nov 02 15:13:03.570783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:03.570747 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" event={"ID":"17d0d0b6-8ac6-4c8a-8784-d509c9838c18","Type":"ContainerStarted","Data":"cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee"} Nov 02 15:13:03.571162 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:03.570794 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" event={"ID":"17d0d0b6-8ac6-4c8a-8784-d509c9838c18","Type":"ContainerStarted","Data":"77c32135c4a1c7937e5110a942e9ef3c2b388a0a0e5390d7cb8ed73e9bbd710d"} Nov 02 15:13:03.941719 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:03.941669 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podStartSLOduration=1.9416553859999999 podStartE2EDuration="1.941655386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:03.940225175 +0000 UTC m=+938.104777716" watchObservedRunningTime="2026-11-02 15:13:03.941655386 +0000 UTC m=+938.106207925" Nov 02 15:13:04.548245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:04.548210 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:05.343367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:05.343330 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:13:05.355857 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:05.355834 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:13:05.591873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:05.591845 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:13:12.548931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:12.548835 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:12.566268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:12.566234 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:12.629637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:12.629606 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:13:26.318830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.318800 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl"] Nov 02 15:13:26.322960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.322935 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.421165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.421129 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"simulator-w5-891723-kserve-self-signed-certs\"" Nov 02 15:13:26.421165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.421138 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"openshift-service-ca.crt\"" Nov 02 15:13:26.421435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.421224 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"default-dockercfg-sw5mb\"" Nov 02 15:13:26.421435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.421241 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-891723\"/\"kube-root-ca.crt\"" Nov 02 15:13:26.423384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.423359 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl"] Nov 02 15:13:26.462781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.462740 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.462781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.462781 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.463026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.462818 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.463026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.462930 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.463026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.462970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.463199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.463074 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlbx\" (UniqueName: \"kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.563833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.563788 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.563848 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlbx\" (UniqueName: \"kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.563965 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.563995 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.564040 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.564161 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564374 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.564349 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.564659 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.564846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.564824 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.566189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.566163 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.566590 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.566574 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.622930 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.622896 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlbx\" (UniqueName: \"kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx\") pod \"simulator-w5-891723-kserve-755b4d8fd4-ltthl\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:26.635643 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:26.635613 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:27.033631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:27.030826 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl"] Nov 02 15:13:27.041707 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:13:27.041676 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4653ab7_34d7_4687_9a0f_9a194ba66cdb.slice/crio-d0d67fc7a0245da4263485953416e14eb00da618ebf6f0e39b4b214ba8c6fd98 WatchSource:0}: Error finding container d0d67fc7a0245da4263485953416e14eb00da618ebf6f0e39b4b214ba8c6fd98: Status 404 returned error can't find the container with id d0d67fc7a0245da4263485953416e14eb00da618ebf6f0e39b4b214ba8c6fd98 Nov 02 15:13:27.703994 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:27.703957 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" event={"ID":"c4653ab7-34d7-4687-9a0f-9a194ba66cdb","Type":"ContainerStarted","Data":"fef7dc727d057df252ebb8e6a31bba3d4258069c098dfcf2f58a37d5d9ae5364"} Nov 02 15:13:27.704411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:27.704006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" event={"ID":"c4653ab7-34d7-4687-9a0f-9a194ba66cdb","Type":"ContainerStarted","Data":"d0d67fc7a0245da4263485953416e14eb00da618ebf6f0e39b4b214ba8c6fd98"} Nov 02 15:13:27.821148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:27.821097 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podStartSLOduration=1.821082619 podStartE2EDuration="1.821082619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:27.819853754 +0000 UTC m=+961.984406294" watchObservedRunningTime="2026-11-02 15:13:27.821082619 +0000 UTC m=+961.985635137" Nov 02 15:13:27.935032 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:13:27.934992 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:13:28.635858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:28.635824 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:36.635995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:36.635962 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:36.648648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:36.648625 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:36.762757 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:36.762727 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:13:38.877254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.877222 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq"] Nov 02 15:13:38.882484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.882463 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:38.981427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.981384 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"simulator-w0-61a835-kserve-self-signed-certs\"" Nov 02 15:13:38.981427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.981384 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"kube-root-ca.crt\"" Nov 02 15:13:38.981427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.981398 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"default-dockercfg-bxzxs\"" Nov 02 15:13:38.981746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.981452 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"openshift-service-ca.crt\"" Nov 02 15:13:38.983661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.983639 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq"] Nov 02 15:13:38.987871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.987851 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:38.987992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.987916 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:38.987992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.987938 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:38.988112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.988087 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:38.988275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.988249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:38.988372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:38.988353 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nmcm\" (UniqueName: \"kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.089600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.089564 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.089600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.089604 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.089805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.089623 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.089805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.089676 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.089805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.089708 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nmcm\" (UniqueName: \"kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.089805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.089742 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.090079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.090056 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.090125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.090074 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.090162 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.090130 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.091981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.091954 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.092120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.092103 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.148613 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.148511 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nmcm\" (UniqueName: \"kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm\") pod \"simulator-w0-61a835-kserve-59dc5db855-5rswq\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.195602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.195563 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:39.572902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.572815 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq"] Nov 02 15:13:39.578702 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:13:39.578676 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66b777c_1cfc_4dff_add8_63e95b1ac41b.slice/crio-0cc111df73ed5bd38cfad8c2f9ebf083b93a420829c0b29ebce639388d065e99 WatchSource:0}: Error finding container 0cc111df73ed5bd38cfad8c2f9ebf083b93a420829c0b29ebce639388d065e99: Status 404 returned error can't find the container with id 0cc111df73ed5bd38cfad8c2f9ebf083b93a420829c0b29ebce639388d065e99 Nov 02 15:13:39.770808 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.770777 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" event={"ID":"f66b777c-1cfc-4dff-add8-63e95b1ac41b","Type":"ContainerStarted","Data":"d77d098b0b63910f1b41263e709650325cb4d09be2db00cf1850b2621b915d0f"} Nov 02 15:13:39.770979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.770818 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" event={"ID":"f66b777c-1cfc-4dff-add8-63e95b1ac41b","Type":"ContainerStarted","Data":"0cc111df73ed5bd38cfad8c2f9ebf083b93a420829c0b29ebce639388d065e99"} Nov 02 15:13:39.891318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:39.891264 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podStartSLOduration=1.891250323 podStartE2EDuration="1.891250323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:13:39.889541826 +0000 UTC m=+974.054094370" watchObservedRunningTime="2026-11-02 15:13:39.891250323 +0000 UTC m=+974.055802862" Nov 02 15:13:40.196278 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:40.196198 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:49.195996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:49.195957 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:49.208472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:49.208445 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:49.836345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:13:49.836314 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:13:57.936675 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:13:57.936601 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:14:02.760334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.760291 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc"] Nov 02 15:14:02.764294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.764271 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:02.863807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.863774 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"premium-w2-fce776-kserve-self-signed-certs\"" Nov 02 15:14:02.865995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.865974 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc"] Nov 02 15:14:02.926418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.926382 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:02.926418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.926420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:02.926645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.926452 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntwtp\" (UniqueName: \"kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:02.926645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.926589 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:02.926713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.926651 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:02.926750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:02.926715 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.027553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027458 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.027553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027497 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.027843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027555 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.027843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.027843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027622 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntwtp\" (UniqueName: \"kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.027843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027700 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.028118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.027912 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.028118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.028001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.028243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.028222 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.030213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.030184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.030312 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.030192 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.086475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.086434 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntwtp\" (UniqueName: \"kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp\") pod \"premium-w2-fce776-kserve-86dc97cd46-s8twc\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.376727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.376695 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:03.750198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.750116 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc"] Nov 02 15:14:03.755932 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:14:03.755903 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8bf04d7_e230_42d5_ad2a_d087d2611533.slice/crio-9555170abcf0584f78f33a54d728902279f02b3e3773e28f0a51029f901dec36 WatchSource:0}: Error finding container 9555170abcf0584f78f33a54d728902279f02b3e3773e28f0a51029f901dec36: Status 404 returned error can't find the container with id 9555170abcf0584f78f33a54d728902279f02b3e3773e28f0a51029f901dec36 Nov 02 15:14:03.907507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.907470 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" event={"ID":"a8bf04d7-e230-42d5-ad2a-d087d2611533","Type":"ContainerStarted","Data":"87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f"} Nov 02 15:14:03.907911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:03.907538 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" event={"ID":"a8bf04d7-e230-42d5-ad2a-d087d2611533","Type":"ContainerStarted","Data":"9555170abcf0584f78f33a54d728902279f02b3e3773e28f0a51029f901dec36"} Nov 02 15:14:04.020756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:04.020661 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podStartSLOduration=2.020645639 podStartE2EDuration="2.020645639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:14:04.019846538 +0000 UTC m=+998.184399082" watchObservedRunningTime="2026-11-02 15:14:04.020645639 +0000 UTC m=+998.185198179" Nov 02 15:14:04.377405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:04.377377 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:13.377359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:13.377324 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:13.390240 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:13.390216 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:13.986768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:14:13.986722 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:14:27.933482 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:14:27.933442 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:14:57.934506 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:14:57.934418 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:15:00.180698 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.180666 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l"] Nov 02 15:15:00.184336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.184318 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.185316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.185298 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6"] Nov 02 15:15:00.188657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.188640 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.218792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.218763 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq"] Nov 02 15:15:00.222202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.222187 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.228673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.228647 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn"] Nov 02 15:15:00.232184 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.232165 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.235748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.235730 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:15:00.279339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.279308 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l"] Nov 02 15:15:00.282128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282101 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6"] Nov 02 15:15:00.282290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282174 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdf4s\" (UniqueName: \"kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s\") pod \"maas-api-key-cleanup-29893875-dv2d6\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.282290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282204 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqwz\" (UniqueName: \"kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.282290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282254 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.282488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282457 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkz5\" (UniqueName: \"kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.282629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282509 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.282629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282580 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6hv\" (UniqueName: \"kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.282756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282635 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-dv2d6\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.282756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.282692 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.321566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.321541 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq"] Nov 02 15:15:00.324464 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.324439 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn"] Nov 02 15:15:00.383688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383657 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.383833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383698 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6hv\" (UniqueName: \"kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.383833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-dv2d6\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.383912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383879 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.383912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383908 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdf4s\" (UniqueName: \"kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s\") pod \"maas-api-key-cleanup-29893875-dv2d6\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.383985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383925 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqwz\" (UniqueName: \"kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.383985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.383968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.384074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.384051 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkz5\" (UniqueName: \"kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.384439 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.384417 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.384678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.384656 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893875-dv2d6\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.384757 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.384656 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.384812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.384756 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.442856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.442784 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdf4s\" (UniqueName: \"kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s\") pod \"maas-api-key-cleanup-29893875-dv2d6\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.443001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.442872 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqwz\" (UniqueName: \"kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz\") pod \"maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.443001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.442910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6hv\" (UniqueName: \"kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.451745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.451721 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkz5\" (UniqueName: \"kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.498761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.498730 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:00.504922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.504904 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:00.535048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.535007 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:00.544742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.544712 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:00.880161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.880133 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l"] Nov 02 15:15:00.885753 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:15:00.885728 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc73d401_fdfc_4476_8279_791e8abd7b2b.slice/crio-6b3bbd8e0a4bf06eaa97b4e9e59cc13214c22d2ca83faa435f7f62d2a0d16847 WatchSource:0}: Error finding container 6b3bbd8e0a4bf06eaa97b4e9e59cc13214c22d2ca83faa435f7f62d2a0d16847: Status 404 returned error can't find the container with id 6b3bbd8e0a4bf06eaa97b4e9e59cc13214c22d2ca83faa435f7f62d2a0d16847 Nov 02 15:15:00.928823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.928801 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq"] Nov 02 15:15:00.931804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.931785 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6"] Nov 02 15:15:00.934664 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:15:00.934641 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750b5a6d_9f01_403c_a017_5027791ed320.slice/crio-eedb60955e67ba04fd9848cc591a1f7da936d0141851f1e6f1fef2787e28b001 WatchSource:0}: Error finding container eedb60955e67ba04fd9848cc591a1f7da936d0141851f1e6f1fef2787e28b001: Status 404 returned error can't find the container with id eedb60955e67ba04fd9848cc591a1f7da936d0141851f1e6f1fef2787e28b001 Nov 02 15:15:00.936882 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:15:00.936859 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83c6163_e26d_4100_ab15_7854bcf25348.slice/crio-f00ed07393de76bebbfb5842ff5cfdc49d9d052ed371db4c2b85320fa6d82ee9 WatchSource:0}: Error finding container f00ed07393de76bebbfb5842ff5cfdc49d9d052ed371db4c2b85320fa6d82ee9: Status 404 returned error can't find the container with id f00ed07393de76bebbfb5842ff5cfdc49d9d052ed371db4c2b85320fa6d82ee9 Nov 02 15:15:00.981819 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:00.981796 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn"] Nov 02 15:15:00.986997 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:15:00.986964 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2e4fe4_e387_43a6_95ff_8338021aebff.slice/crio-914fa6b35f260ce0d321345bf4cab1447baf69acdbba008a57f22bc3d02d50ac WatchSource:0}: Error finding container 914fa6b35f260ce0d321345bf4cab1447baf69acdbba008a57f22bc3d02d50ac: Status 404 returned error can't find the container with id 914fa6b35f260ce0d321345bf4cab1447baf69acdbba008a57f22bc3d02d50ac Nov 02 15:15:01.244561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:01.244443 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" event={"ID":"a83c6163-e26d-4100-ab15-7854bcf25348","Type":"ContainerStarted","Data":"f00ed07393de76bebbfb5842ff5cfdc49d9d052ed371db4c2b85320fa6d82ee9"} Nov 02 15:15:01.245926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:01.245894 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerStarted","Data":"eedb60955e67ba04fd9848cc591a1f7da936d0141851f1e6f1fef2787e28b001"} Nov 02 15:15:01.247136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:01.247109 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" event={"ID":"fc73d401-fdfc-4476-8279-791e8abd7b2b","Type":"ContainerStarted","Data":"6b3bbd8e0a4bf06eaa97b4e9e59cc13214c22d2ca83faa435f7f62d2a0d16847"} Nov 02 15:15:01.248221 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:01.248201 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerStarted","Data":"914fa6b35f260ce0d321345bf4cab1447baf69acdbba008a57f22bc3d02d50ac"} Nov 02 15:15:02.899057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:02.899019 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs"] Nov 02 15:15:02.904321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:02.904298 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:02.958204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:02.958174 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs"] Nov 02 15:15:03.001746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.001720 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-61a835\"/\"premium-w0-61a835-kserve-self-signed-certs\"" Nov 02 15:15:03.023251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.023223 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.023251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.023251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcc8\" (UniqueName: \"kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.023418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.023361 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.023418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.023398 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.023501 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.023495 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.023693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.023671 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125248 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125413 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125502 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125603 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125628 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcc8\" (UniqueName: \"kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.125948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125927 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.126030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.125955 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.127636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.127614 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.127913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.127895 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.194213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.194139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcc8\" (UniqueName: \"kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8\") pod \"premium-w0-61a835-kserve-65d54dcf94-6tkqs\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.217002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.216965 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:03.596118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:03.596043 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs"] Nov 02 15:15:03.601912 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:15:03.601885 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea97b0cb_2f77_4da3_a23c_1a596c2d706d.slice/crio-9c1563dfb1fa208a2a87a53816473011fb65e1a27f08f4ea384feb82d40d5d10 WatchSource:0}: Error finding container 9c1563dfb1fa208a2a87a53816473011fb65e1a27f08f4ea384feb82d40d5d10: Status 404 returned error can't find the container with id 9c1563dfb1fa208a2a87a53816473011fb65e1a27f08f4ea384feb82d40d5d10 Nov 02 15:15:04.276772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:04.275878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" event={"ID":"ea97b0cb-2f77-4da3-a23c-1a596c2d706d","Type":"ContainerStarted","Data":"c807b53a2c7699021117f3f29bb837d58d1c615c28e57829460b551ff11bc296"} Nov 02 15:15:04.276772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:04.275930 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" event={"ID":"ea97b0cb-2f77-4da3-a23c-1a596c2d706d","Type":"ContainerStarted","Data":"9c1563dfb1fa208a2a87a53816473011fb65e1a27f08f4ea384feb82d40d5d10"} Nov 02 15:15:04.394506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:04.394431 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podStartSLOduration=2.394416077 podStartE2EDuration="2.394416077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:04.391260356 +0000 UTC m=+1058.555812896" watchObservedRunningTime="2026-11-02 15:15:04.394416077 +0000 UTC m=+1058.558968617" Nov 02 15:15:05.217107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:05.217061 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:06.298481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:06.298446 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" event={"ID":"a83c6163-e26d-4100-ab15-7854bcf25348","Type":"ContainerStarted","Data":"18f32a6d029bff13ad6f83e211af810ee96e98482caeab27a615453493bb4686"} Nov 02 15:15:06.412238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:06.412167 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" podStartSLOduration=1.625056985 podStartE2EDuration="6.412149642s" totalImagesPullingTime="4.787092657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:06.409559106 +0000 UTC m=+1060.574111643" watchObservedRunningTime="2026-11-02 15:15:06.412149642 +0000 UTC m=+1060.576702183" Nov 02 15:15:07.306314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.306216 5091 generic.go:356] "Generic (PLEG): container finished" podID="a83c6163-e26d-4100-ab15-7854bcf25348" containerID="18f32a6d029bff13ad6f83e211af810ee96e98482caeab27a615453493bb4686" exitCode=60 Nov 02 15:15:07.306314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.306294 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" event={"ID":"a83c6163-e26d-4100-ab15-7854bcf25348","Type":"ContainerDied","Data":"18f32a6d029bff13ad6f83e211af810ee96e98482caeab27a615453493bb4686"} Nov 02 15:15:07.306847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.306593 5091 scope.go:122] "RemoveContainer" containerID="18f32a6d029bff13ad6f83e211af810ee96e98482caeab27a615453493bb4686" Nov 02 15:15:07.308129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.308104 5091 generic.go:356] "Generic (PLEG): container finished" podID="750b5a6d-9f01-403c-a017-5027791ed320" containerID="1efb27cbfa54c5d9caecc083f55505e2479662497c255fbb1cb3ab6bbe643804" exitCode=60 Nov 02 15:15:07.308248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.308206 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerDied","Data":"1efb27cbfa54c5d9caecc083f55505e2479662497c255fbb1cb3ab6bbe643804"} Nov 02 15:15:07.308387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.308370 5091 scope.go:122] "RemoveContainer" containerID="1efb27cbfa54c5d9caecc083f55505e2479662497c255fbb1cb3ab6bbe643804" Nov 02 15:15:07.310565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.310546 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerID="ef5d2a206212c739f11416a10c1fb51fe9c22722e7bcbff428fd18ff9c52ead8" exitCode=60 Nov 02 15:15:07.310642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.310578 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" event={"ID":"fc73d401-fdfc-4476-8279-791e8abd7b2b","Type":"ContainerDied","Data":"ef5d2a206212c739f11416a10c1fb51fe9c22722e7bcbff428fd18ff9c52ead8"} Nov 02 15:15:07.310798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:07.310782 5091 scope.go:122] "RemoveContainer" containerID="ef5d2a206212c739f11416a10c1fb51fe9c22722e7bcbff428fd18ff9c52ead8" Nov 02 15:15:08.317508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.317480 5091 generic.go:356] "Generic (PLEG): container finished" podID="a83c6163-e26d-4100-ab15-7854bcf25348" containerID="3bac1a2be03c29ea74cf98059a965ed3625d293809d22db2e308a216fff28870" exitCode=60 Nov 02 15:15:08.317905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.317567 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" event={"ID":"a83c6163-e26d-4100-ab15-7854bcf25348","Type":"ContainerDied","Data":"3bac1a2be03c29ea74cf98059a965ed3625d293809d22db2e308a216fff28870"} Nov 02 15:15:08.317905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.317630 5091 scope.go:122] "RemoveContainer" containerID="18f32a6d029bff13ad6f83e211af810ee96e98482caeab27a615453493bb4686" Nov 02 15:15:08.317905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.317872 5091 scope.go:122] "RemoveContainer" containerID="3bac1a2be03c29ea74cf98059a965ed3625d293809d22db2e308a216fff28870" Nov 02 15:15:08.318158 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:08.318133 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-dv2d6_odh-ai-gateway-infra(a83c6163-e26d-4100-ab15-7854bcf25348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" Nov 02 15:15:08.319433 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.319412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerStarted","Data":"955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f"} Nov 02 15:15:08.321142 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.321120 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerID="df6ad96f1624e5d0e5b54884d07ef10024ee996fffd4765cee76c5dd3136586b" exitCode=60 Nov 02 15:15:08.321250 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.321199 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" event={"ID":"fc73d401-fdfc-4476-8279-791e8abd7b2b","Type":"ContainerDied","Data":"df6ad96f1624e5d0e5b54884d07ef10024ee996fffd4765cee76c5dd3136586b"} Nov 02 15:15:08.321435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.321418 5091 scope.go:122] "RemoveContainer" containerID="df6ad96f1624e5d0e5b54884d07ef10024ee996fffd4765cee76c5dd3136586b" Nov 02 15:15:08.321675 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:08.321652 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l_odh-ai-gateway-infra(fc73d401-fdfc-4476-8279-791e8abd7b2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" Nov 02 15:15:08.322688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.322667 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerStarted","Data":"a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307"} Nov 02 15:15:08.406345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.406316 5091 scope.go:122] "RemoveContainer" containerID="ef5d2a206212c739f11416a10c1fb51fe9c22722e7bcbff428fd18ff9c52ead8" Nov 02 15:15:08.540966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.540878 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" podStartSLOduration=3.208368709 podStartE2EDuration="8.540863748s" totalImagesPullingTime="5.332495039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:08.538830508 +0000 UTC m=+1062.703383048" watchObservedRunningTime="2026-11-02 15:15:08.540863748 +0000 UTC m=+1062.705416340" Nov 02 15:15:08.651294 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:08.651259 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2e4fe4_e387_43a6_95ff_8338021aebff.slice/crio-conmon-a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2e4fe4_e387_43a6_95ff_8338021aebff.slice/crio-a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:15:08.761379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:08.761320 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" podStartSLOduration=2.027780853 podStartE2EDuration="8.761305152s" totalImagesPullingTime="6.733524299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:08.760131705 +0000 UTC m=+1062.924684247" watchObservedRunningTime="2026-11-02 15:15:08.761305152 +0000 UTC m=+1062.925858074" Nov 02 15:15:09.332261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.332222 5091 scope.go:122] "RemoveContainer" containerID="3bac1a2be03c29ea74cf98059a965ed3625d293809d22db2e308a216fff28870" Nov 02 15:15:09.332795 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:09.332487 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-dv2d6_odh-ai-gateway-infra(a83c6163-e26d-4100-ab15-7854bcf25348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" Nov 02 15:15:09.334041 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.334011 5091 generic.go:356] "Generic (PLEG): container finished" podID="750b5a6d-9f01-403c-a017-5027791ed320" containerID="955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f" exitCode=60 Nov 02 15:15:09.334159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.334092 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerDied","Data":"955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f"} Nov 02 15:15:09.334213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.334194 5091 scope.go:122] "RemoveContainer" containerID="1efb27cbfa54c5d9caecc083f55505e2479662497c255fbb1cb3ab6bbe643804" Nov 02 15:15:09.334306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.334289 5091 scope.go:122] "RemoveContainer" containerID="955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f" Nov 02 15:15:09.334582 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:09.334558 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq_odh-ai-gateway-infra(750b5a6d-9f01-403c-a017-5027791ed320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" podUID="750b5a6d-9f01-403c-a017-5027791ed320" Nov 02 15:15:09.336052 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.336030 5091 scope.go:122] "RemoveContainer" containerID="df6ad96f1624e5d0e5b54884d07ef10024ee996fffd4765cee76c5dd3136586b" Nov 02 15:15:09.336282 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:09.336261 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l_odh-ai-gateway-infra(fc73d401-fdfc-4476-8279-791e8abd7b2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" Nov 02 15:15:09.337338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.337315 5091 generic.go:356] "Generic (PLEG): container finished" podID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerID="a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307" exitCode=60 Nov 02 15:15:09.337436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.337404 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerDied","Data":"a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307"} Nov 02 15:15:09.337650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:09.337633 5091 scope.go:122] "RemoveContainer" containerID="a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307" Nov 02 15:15:10.347091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:10.347062 5091 scope.go:122] "RemoveContainer" containerID="955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f" Nov 02 15:15:10.347642 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:10.347318 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq_odh-ai-gateway-infra(750b5a6d-9f01-403c-a017-5027791ed320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" podUID="750b5a6d-9f01-403c-a017-5027791ed320" Nov 02 15:15:10.350279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:10.350248 5091 generic.go:356] "Generic (PLEG): container finished" podID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerID="63f3821e30554feb058902bf1ea97967416643c77ce11dee5f71ac048e168317" exitCode=60 Nov 02 15:15:10.350375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:10.350294 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerDied","Data":"63f3821e30554feb058902bf1ea97967416643c77ce11dee5f71ac048e168317"} Nov 02 15:15:10.350375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:10.350347 5091 scope.go:122] "RemoveContainer" containerID="a250c6c58db058e47ed1cbb57225f6ede04e73b816db67d78badc44856432307" Nov 02 15:15:10.350602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:10.350582 5091 scope.go:122] "RemoveContainer" containerID="63f3821e30554feb058902bf1ea97967416643c77ce11dee5f71ac048e168317" Nov 02 15:15:10.350868 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:10.350842 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn_odh-ai-gateway-infra(2f2e4fe4-e387-43a6-95ff-8338021aebff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" Nov 02 15:15:11.358341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:11.358316 5091 scope.go:122] "RemoveContainer" containerID="63f3821e30554feb058902bf1ea97967416643c77ce11dee5f71ac048e168317" Nov 02 15:15:11.358727 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:11.358573 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn_odh-ai-gateway-infra(2f2e4fe4-e387-43a6-95ff-8338021aebff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" Nov 02 15:15:13.217831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:13.217797 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:13.230429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:13.230401 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:13.381261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:13.381234 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:15:19.952629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:19.952596 5091 scope.go:122] "RemoveContainer" containerID="955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f" Nov 02 15:15:20.410816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:20.410780 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerStarted","Data":"a1c1215f9f64463ccb55bacfb8623351804e494cbe0389997a9dc626fd97b427"} Nov 02 15:15:21.417894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.417856 5091 generic.go:356] "Generic (PLEG): container finished" podID="750b5a6d-9f01-403c-a017-5027791ed320" containerID="a1c1215f9f64463ccb55bacfb8623351804e494cbe0389997a9dc626fd97b427" exitCode=60 Nov 02 15:15:21.418348 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.417929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerDied","Data":"a1c1215f9f64463ccb55bacfb8623351804e494cbe0389997a9dc626fd97b427"} Nov 02 15:15:21.418348 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.417989 5091 scope.go:122] "RemoveContainer" containerID="955e509f11076dd59dc21d262896dadb443cb5cef7b116e8e0a577f74bf00d5f" Nov 02 15:15:21.418348 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.418297 5091 scope.go:122] "RemoveContainer" containerID="a1c1215f9f64463ccb55bacfb8623351804e494cbe0389997a9dc626fd97b427" Nov 02 15:15:21.418619 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:21.418594 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq_odh-ai-gateway-infra(750b5a6d-9f01-403c-a017-5027791ed320)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" podUID="750b5a6d-9f01-403c-a017-5027791ed320" Nov 02 15:15:21.534745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.534712 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq"] Nov 02 15:15:21.953276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.953245 5091 scope.go:122] "RemoveContainer" containerID="df6ad96f1624e5d0e5b54884d07ef10024ee996fffd4765cee76c5dd3136586b" Nov 02 15:15:21.953450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:21.953354 5091 scope.go:122] "RemoveContainer" containerID="3bac1a2be03c29ea74cf98059a965ed3625d293809d22db2e308a216fff28870" Nov 02 15:15:22.427241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.427212 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" event={"ID":"fc73d401-fdfc-4476-8279-791e8abd7b2b","Type":"ContainerStarted","Data":"3a92bf165002e5c8120667c3c0069f538f72b7957919cbe45c0c79eaca93448f"} Nov 02 15:15:22.572125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.572091 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:22.636804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.636776 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca\") pod \"750b5a6d-9f01-403c-a017-5027791ed320\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " Nov 02 15:15:22.636960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.636895 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5\" (UniqueName: \"kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5\") pod \"750b5a6d-9f01-403c-a017-5027791ed320\" (UID: \"750b5a6d-9f01-403c-a017-5027791ed320\") " Nov 02 15:15:22.637154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.637130 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca" pod "750b5a6d-9f01-403c-a017-5027791ed320" (UID: "750b5a6d-9f01-403c-a017-5027791ed320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:22.638985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.638964 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5" pod "750b5a6d-9f01-403c-a017-5027791ed320" (UID: "750b5a6d-9f01-403c-a017-5027791ed320"). InnerVolumeSpecName "kube-api-access-7fkz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:22.649151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.649110 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" podStartSLOduration=17.652129714 podStartE2EDuration="22.649098854s" totalImagesPullingTime="4.99696914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:22.646644149 +0000 UTC m=+1076.811196684" watchObservedRunningTime="2026-11-02 15:15:22.649098854 +0000 UTC m=+1076.813651441" Nov 02 15:15:22.737769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.737736 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/750b5a6d-9f01-403c-a017-5027791ed320-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:22.737769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:22.737764 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkz5\" (UniqueName: \"kubernetes.io/projected/750b5a6d-9f01-403c-a017-5027791ed320-kube-api-access-7fkz5\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:23.434355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.434319 5091 generic.go:356] "Generic (PLEG): container finished" podID="a83c6163-e26d-4100-ab15-7854bcf25348" containerID="a195458e5779088817e9e142543f9a2355dd08cb92666ce95845de9574810527" exitCode=60 Nov 02 15:15:23.434830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.434395 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" event={"ID":"a83c6163-e26d-4100-ab15-7854bcf25348","Type":"ContainerDied","Data":"a195458e5779088817e9e142543f9a2355dd08cb92666ce95845de9574810527"} Nov 02 15:15:23.434830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.434488 5091 scope.go:122] "RemoveContainer" containerID="3bac1a2be03c29ea74cf98059a965ed3625d293809d22db2e308a216fff28870" Nov 02 15:15:23.434957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.434898 5091 scope.go:122] "RemoveContainer" containerID="a195458e5779088817e9e142543f9a2355dd08cb92666ce95845de9574810527" Nov 02 15:15:23.435194 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:23.435166 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893875-dv2d6_odh-ai-gateway-infra(a83c6163-e26d-4100-ab15-7854bcf25348)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" Nov 02 15:15:23.435959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.435777 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" Nov 02 15:15:23.435959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.435812 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq" event={"ID":"750b5a6d-9f01-403c-a017-5027791ed320","Type":"ContainerDied","Data":"eedb60955e67ba04fd9848cc591a1f7da936d0141851f1e6f1fef2787e28b001"} Nov 02 15:15:23.437969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.437947 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerID="3a92bf165002e5c8120667c3c0069f538f72b7957919cbe45c0c79eaca93448f" exitCode=60 Nov 02 15:15:23.438065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.437997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" event={"ID":"fc73d401-fdfc-4476-8279-791e8abd7b2b","Type":"ContainerDied","Data":"3a92bf165002e5c8120667c3c0069f538f72b7957919cbe45c0c79eaca93448f"} Nov 02 15:15:23.438365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.438340 5091 scope.go:122] "RemoveContainer" containerID="3a92bf165002e5c8120667c3c0069f538f72b7957919cbe45c0c79eaca93448f" Nov 02 15:15:23.438613 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:23.438590 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l_odh-ai-gateway-infra(fc73d401-fdfc-4476-8279-791e8abd7b2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" Nov 02 15:15:23.447619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.447603 5091 scope.go:122] "RemoveContainer" containerID="a1c1215f9f64463ccb55bacfb8623351804e494cbe0389997a9dc626fd97b427" Nov 02 15:15:23.456011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.455991 5091 scope.go:122] "RemoveContainer" containerID="df6ad96f1624e5d0e5b54884d07ef10024ee996fffd4765cee76c5dd3136586b" Nov 02 15:15:23.658017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.657985 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l"] Nov 02 15:15:23.675108 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.675079 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq"] Nov 02 15:15:23.679732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.679712 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893875-468nq"] Nov 02 15:15:23.953186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.953156 5091 scope.go:122] "RemoveContainer" containerID="63f3821e30554feb058902bf1ea97967416643c77ce11dee5f71ac048e168317" Nov 02 15:15:23.961191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:23.961163 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750b5a6d-9f01-403c-a017-5027791ed320" path="/var/lib/kubelet/pods/750b5a6d-9f01-403c-a017-5027791ed320/volumes" Nov 02 15:15:24.454333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.454295 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerStarted","Data":"249cec9a276012cd1fca57595bb3ac03998990c51cdf32827fdeee89891a5fa6"} Nov 02 15:15:24.577639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.577615 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6"] Nov 02 15:15:24.591631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.591606 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:24.655306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.655274 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca\") pod \"fc73d401-fdfc-4476-8279-791e8abd7b2b\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " Nov 02 15:15:24.655464 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.655382 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz\" (UniqueName: \"kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz\") pod \"fc73d401-fdfc-4476-8279-791e8abd7b2b\" (UID: \"fc73d401-fdfc-4476-8279-791e8abd7b2b\") " Nov 02 15:15:24.655692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.655669 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca" pod "fc73d401-fdfc-4476-8279-791e8abd7b2b" (UID: "fc73d401-fdfc-4476-8279-791e8abd7b2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:24.655829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.655813 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc73d401-fdfc-4476-8279-791e8abd7b2b-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.657372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.657348 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz" pod "fc73d401-fdfc-4476-8279-791e8abd7b2b" (UID: "fc73d401-fdfc-4476-8279-791e8abd7b2b"). InnerVolumeSpecName "kube-api-access-dwqwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:24.732937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.732910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:24.756824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.756802 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s\" (UniqueName: \"kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s\") pod \"a83c6163-e26d-4100-ab15-7854bcf25348\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " Nov 02 15:15:24.756918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.756842 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca\") pod \"a83c6163-e26d-4100-ab15-7854bcf25348\" (UID: \"a83c6163-e26d-4100-ab15-7854bcf25348\") " Nov 02 15:15:24.757162 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.757142 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqwz\" (UniqueName: \"kubernetes.io/projected/fc73d401-fdfc-4476-8279-791e8abd7b2b-kube-api-access-dwqwz\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.757249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.757234 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca" pod "a83c6163-e26d-4100-ab15-7854bcf25348" (UID: "a83c6163-e26d-4100-ab15-7854bcf25348"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:24.758958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.758906 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s" pod "a83c6163-e26d-4100-ab15-7854bcf25348" (UID: "a83c6163-e26d-4100-ab15-7854bcf25348"). InnerVolumeSpecName "kube-api-access-hdf4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:24.858183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.858159 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdf4s\" (UniqueName: \"kubernetes.io/projected/a83c6163-e26d-4100-ab15-7854bcf25348-kube-api-access-hdf4s\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:24.858183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:24.858183 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a83c6163-e26d-4100-ab15-7854bcf25348-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:25.461216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.461189 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" Nov 02 15:15:25.461729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.461184 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l" event={"ID":"fc73d401-fdfc-4476-8279-791e8abd7b2b","Type":"ContainerDied","Data":"6b3bbd8e0a4bf06eaa97b4e9e59cc13214c22d2ca83faa435f7f62d2a0d16847"} Nov 02 15:15:25.461729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.461352 5091 scope.go:122] "RemoveContainer" containerID="3a92bf165002e5c8120667c3c0069f538f72b7957919cbe45c0c79eaca93448f" Nov 02 15:15:25.463656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.463629 5091 generic.go:356] "Generic (PLEG): container finished" podID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerID="249cec9a276012cd1fca57595bb3ac03998990c51cdf32827fdeee89891a5fa6" exitCode=60 Nov 02 15:15:25.463773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.463705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerDied","Data":"249cec9a276012cd1fca57595bb3ac03998990c51cdf32827fdeee89891a5fa6"} Nov 02 15:15:25.464114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.464091 5091 scope.go:122] "RemoveContainer" containerID="249cec9a276012cd1fca57595bb3ac03998990c51cdf32827fdeee89891a5fa6" Nov 02 15:15:25.464370 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:25.464345 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn_odh-ai-gateway-infra(2f2e4fe4-e387-43a6-95ff-8338021aebff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" Nov 02 15:15:25.465435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.465403 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" event={"ID":"a83c6163-e26d-4100-ab15-7854bcf25348","Type":"ContainerDied","Data":"f00ed07393de76bebbfb5842ff5cfdc49d9d052ed371db4c2b85320fa6d82ee9"} Nov 02 15:15:25.465574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.465416 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6" Nov 02 15:15:25.473082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.473063 5091 scope.go:122] "RemoveContainer" containerID="63f3821e30554feb058902bf1ea97967416643c77ce11dee5f71ac048e168317" Nov 02 15:15:25.485951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.485934 5091 scope.go:122] "RemoveContainer" containerID="a195458e5779088817e9e142543f9a2355dd08cb92666ce95845de9574810527" Nov 02 15:15:25.688985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.688954 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn"] Nov 02 15:15:25.739254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.736991 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6"] Nov 02 15:15:25.747154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.747114 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893875-dv2d6"] Nov 02 15:15:25.849426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.849391 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l"] Nov 02 15:15:25.853948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.853922 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8a2e6f-29893875-7rv9l"] Nov 02 15:15:25.959922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.959888 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" path="/var/lib/kubelet/pods/a83c6163-e26d-4100-ab15-7854bcf25348/volumes" Nov 02 15:15:25.960374 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:25.960356 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" path="/var/lib/kubelet/pods/fc73d401-fdfc-4476-8279-791e8abd7b2b/volumes" Nov 02 15:15:26.613173 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.613148 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:26.674440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.674404 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv\" (UniqueName: \"kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv\") pod \"2f2e4fe4-e387-43a6-95ff-8338021aebff\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " Nov 02 15:15:26.674612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.674451 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca\") pod \"2f2e4fe4-e387-43a6-95ff-8338021aebff\" (UID: \"2f2e4fe4-e387-43a6-95ff-8338021aebff\") " Nov 02 15:15:26.674847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.674826 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca" pod "2f2e4fe4-e387-43a6-95ff-8338021aebff" (UID: "2f2e4fe4-e387-43a6-95ff-8338021aebff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:26.676493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.676471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv" pod "2f2e4fe4-e387-43a6-95ff-8338021aebff" (UID: "2f2e4fe4-e387-43a6-95ff-8338021aebff"). InnerVolumeSpecName "kube-api-access-sv6hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:26.775766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.775683 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6hv\" (UniqueName: \"kubernetes.io/projected/2f2e4fe4-e387-43a6-95ff-8338021aebff-kube-api-access-sv6hv\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:26.775766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:26.775711 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2e4fe4-e387-43a6-95ff-8338021aebff-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:27.480846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:27.480820 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" Nov 02 15:15:27.481015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:27.480843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn" event={"ID":"2f2e4fe4-e387-43a6-95ff-8338021aebff","Type":"ContainerDied","Data":"914fa6b35f260ce0d321345bf4cab1447baf69acdbba008a57f22bc3d02d50ac"} Nov 02 15:15:27.481015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:27.480902 5091 scope.go:122] "RemoveContainer" containerID="249cec9a276012cd1fca57595bb3ac03998990c51cdf32827fdeee89891a5fa6" Nov 02 15:15:27.637969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:27.637932 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn"] Nov 02 15:15:27.685160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:27.685121 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893875-7vhvn"] Nov 02 15:15:27.934296 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:27.934259 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:15:27.959474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:27.959445 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" path="/var/lib/kubelet/pods/2f2e4fe4-e387-43a6-95ff-8338021aebff/volumes" Nov 02 15:15:30.768975 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.768943 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw"] Nov 02 15:15:30.769403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769390 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769405 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769419 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769423 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769430 5091 cpu_manager.go:414] "containerMap: removing container" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769435 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769442 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769446 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769455 5091 cpu_manager.go:414] "containerMap: removing container" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:15:30.769462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769459 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769472 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769476 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769487 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769491 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769504 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769508 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769530 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769535 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769610 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769617 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769625 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769630 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc73d401-fdfc-4476-8279-791e8abd7b2b" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769635 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769641 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769647 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769652 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769658 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769664 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:15:30.769772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.769669 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:15:30.774203 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.774185 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.813950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.813916 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.814055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.814033 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lpr\" (UniqueName: \"kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.814105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.814093 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.829088 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.829063 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw"] Nov 02 15:15:30.873110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.873083 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:15:30.873251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.873122 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:15:30.914935 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.914901 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lpr\" (UniqueName: \"kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.915087 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.914968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.915087 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.915041 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.915744 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.915724 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.917305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.917289 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:30.973308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:30.973283 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lpr\" (UniqueName: \"kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr\") pod \"maas-api-revoke-sub-2c154634-kbbhw\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:31.088036 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:31.087961 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:31.464937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:31.464904 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw"] Nov 02 15:15:31.470157 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:15:31.470128 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14cbee7_9491_4dab_98d2_77eca7078839.slice/crio-acd21aaf7c8cda32e90ab83f744c614fa9fb42d03c7c1e2bcf1e0f5a4902d49c WatchSource:0}: Error finding container acd21aaf7c8cda32e90ab83f744c614fa9fb42d03c7c1e2bcf1e0f5a4902d49c: Status 404 returned error can't find the container with id acd21aaf7c8cda32e90ab83f744c614fa9fb42d03c7c1e2bcf1e0f5a4902d49c Nov 02 15:15:31.504130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:31.504106 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" event={"ID":"c14cbee7-9491-4dab-98d2-77eca7078839","Type":"ContainerStarted","Data":"acd21aaf7c8cda32e90ab83f744c614fa9fb42d03c7c1e2bcf1e0f5a4902d49c"} Nov 02 15:15:32.510948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:32.510859 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" event={"ID":"c14cbee7-9491-4dab-98d2-77eca7078839","Type":"ContainerStarted","Data":"ffa64e352357f35721b28314f9da5babacd6d1fe12b4dca0b92c83eed6dabb98"} Nov 02 15:15:32.622682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:32.622636 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" podStartSLOduration=2.622622026 podStartE2EDuration="2.622622026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:15:32.622026494 +0000 UTC m=+1086.786579037" watchObservedRunningTime="2026-11-02 15:15:32.622622026 +0000 UTC m=+1086.787174565" Nov 02 15:15:33.518752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:33.518715 5091 generic.go:356] "Generic (PLEG): container finished" podID="c14cbee7-9491-4dab-98d2-77eca7078839" containerID="ffa64e352357f35721b28314f9da5babacd6d1fe12b4dca0b92c83eed6dabb98" exitCode=0 Nov 02 15:15:33.519147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:33.518797 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" event={"ID":"c14cbee7-9491-4dab-98d2-77eca7078839","Type":"ContainerDied","Data":"ffa64e352357f35721b28314f9da5babacd6d1fe12b4dca0b92c83eed6dabb98"} Nov 02 15:15:34.658571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.658542 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:34.756351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.756322 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca\") pod \"c14cbee7-9491-4dab-98d2-77eca7078839\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " Nov 02 15:15:34.756535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.756382 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr\" (UniqueName: \"kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr\") pod \"c14cbee7-9491-4dab-98d2-77eca7078839\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " Nov 02 15:15:34.756535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.756480 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca\") pod \"c14cbee7-9491-4dab-98d2-77eca7078839\" (UID: \"c14cbee7-9491-4dab-98d2-77eca7078839\") " Nov 02 15:15:34.756895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.756861 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca" pod "c14cbee7-9491-4dab-98d2-77eca7078839" (UID: "c14cbee7-9491-4dab-98d2-77eca7078839"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:15:34.758496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.758472 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca" pod "c14cbee7-9491-4dab-98d2-77eca7078839" (UID: "c14cbee7-9491-4dab-98d2-77eca7078839"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:15:34.758627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.758494 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr" pod "c14cbee7-9491-4dab-98d2-77eca7078839" (UID: "c14cbee7-9491-4dab-98d2-77eca7078839"). InnerVolumeSpecName "kube-api-access-f7lpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:15:34.858320 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.858214 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:34.858320 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.858242 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7lpr\" (UniqueName: \"kubernetes.io/projected/c14cbee7-9491-4dab-98d2-77eca7078839-kube-api-access-f7lpr\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:34.858320 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:34.858252 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c14cbee7-9491-4dab-98d2-77eca7078839-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:15:35.531273 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:35.531240 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" Nov 02 15:15:35.531474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:35.531271 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw" event={"ID":"c14cbee7-9491-4dab-98d2-77eca7078839","Type":"ContainerDied","Data":"acd21aaf7c8cda32e90ab83f744c614fa9fb42d03c7c1e2bcf1e0f5a4902d49c"} Nov 02 15:15:35.531474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:15:35.531333 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd21aaf7c8cda32e90ab83f744c614fa9fb42d03c7c1e2bcf1e0f5a4902d49c" Nov 02 15:15:57.935700 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:15:57.935660 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:16:22.371610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.371567 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt"] Nov 02 15:16:22.372242 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372222 5091 cpu_manager.go:414] "containerMap: removing container" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:16:22.372331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372242 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:16:22.372331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372265 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:16:22.372331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372272 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83c6163-e26d-4100-ab15-7854bcf25348" containerName="cleanup" Nov 02 15:16:22.372331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372309 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c14cbee7-9491-4dab-98d2-77eca7078839" containerName="revoke-keys" Nov 02 15:16:22.372331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372317 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c14cbee7-9491-4dab-98d2-77eca7078839" containerName="revoke-keys" Nov 02 15:16:22.372592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372343 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:16:22.372592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372350 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2e4fe4-e387-43a6-95ff-8338021aebff" containerName="cleanup" Nov 02 15:16:22.372592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372456 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="750b5a6d-9f01-403c-a017-5027791ed320" containerName="cleanup" Nov 02 15:16:22.372592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.372474 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c14cbee7-9491-4dab-98d2-77eca7078839" containerName="revoke-keys" Nov 02 15:16:22.376502 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.376486 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.429148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.429112 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt"] Nov 02 15:16:22.472789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.472756 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:22.472963 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.472894 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:22.528538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.528482 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cw4t\" (UniqueName: \"kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.528538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.528532 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.528763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.528612 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.629892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.629797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cw4t\" (UniqueName: \"kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.629892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.629849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.630132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.629901 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.630561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.630536 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.632696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.632669 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.688890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.688862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cw4t\" (UniqueName: \"kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t\") pod \"maas-api-revoke-sub-b88fef46-bbswt\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:22.690596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:22.690573 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:23.069953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:23.069869 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt"] Nov 02 15:16:23.075697 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:23.075666 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708fd46c_763b_43fd_a6da_e50eaf3a8271.slice/crio-43b0005ff813c1e669db9123d402a2cd40ad2d0f989b18cd27a9bdb6658c0541 WatchSource:0}: Error finding container 43b0005ff813c1e669db9123d402a2cd40ad2d0f989b18cd27a9bdb6658c0541: Status 404 returned error can't find the container with id 43b0005ff813c1e669db9123d402a2cd40ad2d0f989b18cd27a9bdb6658c0541 Nov 02 15:16:23.815210 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:23.815170 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" event={"ID":"708fd46c-763b-43fd-a6da-e50eaf3a8271","Type":"ContainerStarted","Data":"ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6"} Nov 02 15:16:23.815643 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:23.815234 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" event={"ID":"708fd46c-763b-43fd-a6da-e50eaf3a8271","Type":"ContainerStarted","Data":"43b0005ff813c1e669db9123d402a2cd40ad2d0f989b18cd27a9bdb6658c0541"} Nov 02 15:16:23.927797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:23.927739 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" podStartSLOduration=1.927719892 podStartE2EDuration="1.927719892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:23.925800298 +0000 UTC m=+1138.090352838" watchObservedRunningTime="2026-11-02 15:16:23.927719892 +0000 UTC m=+1138.092272433" Nov 02 15:16:24.822638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:24.822607 5091 generic.go:356] "Generic (PLEG): container finished" podID="708fd46c-763b-43fd-a6da-e50eaf3a8271" containerID="ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6" exitCode=0 Nov 02 15:16:24.823050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:24.822649 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" event={"ID":"708fd46c-763b-43fd-a6da-e50eaf3a8271","Type":"ContainerDied","Data":"ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6"} Nov 02 15:16:26.009176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.009150 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:26.174157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.174122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t\" (UniqueName: \"kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t\") pod \"708fd46c-763b-43fd-a6da-e50eaf3a8271\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " Nov 02 15:16:26.174364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.174178 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca\") pod \"708fd46c-763b-43fd-a6da-e50eaf3a8271\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " Nov 02 15:16:26.174364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.174241 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca\") pod \"708fd46c-763b-43fd-a6da-e50eaf3a8271\" (UID: \"708fd46c-763b-43fd-a6da-e50eaf3a8271\") " Nov 02 15:16:26.174698 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.174670 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca" pod "708fd46c-763b-43fd-a6da-e50eaf3a8271" (UID: "708fd46c-763b-43fd-a6da-e50eaf3a8271"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:26.176392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.176367 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca" pod "708fd46c-763b-43fd-a6da-e50eaf3a8271" (UID: "708fd46c-763b-43fd-a6da-e50eaf3a8271"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:26.176479 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.176459 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t" pod "708fd46c-763b-43fd-a6da-e50eaf3a8271" (UID: "708fd46c-763b-43fd-a6da-e50eaf3a8271"). InnerVolumeSpecName "kube-api-access-7cw4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:26.276106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.276061 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:26.276106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.276105 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cw4t\" (UniqueName: \"kubernetes.io/projected/708fd46c-763b-43fd-a6da-e50eaf3a8271-kube-api-access-7cw4t\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:26.276317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.276123 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/708fd46c-763b-43fd-a6da-e50eaf3a8271-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:26.840476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.840444 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" Nov 02 15:16:26.840476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.840455 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt" event={"ID":"708fd46c-763b-43fd-a6da-e50eaf3a8271","Type":"ContainerDied","Data":"43b0005ff813c1e669db9123d402a2cd40ad2d0f989b18cd27a9bdb6658c0541"} Nov 02 15:16:26.840722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:26.840541 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43b0005ff813c1e669db9123d402a2cd40ad2d0f989b18cd27a9bdb6658c0541" Nov 02 15:16:27.378167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.378122 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6"] Nov 02 15:16:27.378706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.378625 5091 cpu_manager.go:414] "containerMap: removing container" podUID="708fd46c-763b-43fd-a6da-e50eaf3a8271" containerName="revoke-keys" Nov 02 15:16:27.378706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.378638 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="708fd46c-763b-43fd-a6da-e50eaf3a8271" containerName="revoke-keys" Nov 02 15:16:27.378828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.378747 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="708fd46c-763b-43fd-a6da-e50eaf3a8271" containerName="revoke-keys" Nov 02 15:16:27.385172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.385136 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.438120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.438087 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6"] Nov 02 15:16:27.481883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.481851 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:27.482209 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.482187 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:27.484800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.484777 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb"] Nov 02 15:16:27.487462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.487437 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.487595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.487499 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.487655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.487605 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhqj\" (UniqueName: \"kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.488858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.488837 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.543606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.543571 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8a2e6f\"/\"distinct-w4-8a2e6f-kserve-self-signed-certs\"" Nov 02 15:16:27.543852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.543825 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb"] Nov 02 15:16:27.589223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.589426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589264 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.589426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589343 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhqj\" (UniqueName: \"kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.589426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.589426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589410 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.589646 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589572 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.589700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589689 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.589753 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589740 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.589848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.589827 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqp7\" (UniqueName: \"kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.590448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.590428 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.591689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.591669 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.649333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.649255 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhqj\" (UniqueName: \"kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj\") pod \"maas-api-revoke-sub-47a351ad-2w2n6\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.691486 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691437 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.691711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691585 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.691711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691661 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqp7\" (UniqueName: \"kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.691830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691720 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.691830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691766 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.691830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691800 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.691984 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.691941 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.692354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.692305 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.692528 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.692494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.694644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.694617 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.694867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.694845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.699423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.699398 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:27.761157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.759912 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqp7\" (UniqueName: \"kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7\") pod \"distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.801845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:27.801808 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:27.951133 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:16:27.951031 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:16:28.094528 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.094473 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6"] Nov 02 15:16:28.104783 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:28.104749 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08d279e_fe25_49da_ab6c_8d83c0894b8e.slice/crio-160fb5c6367c72ba36757b0cb3591e0581b568e5814ca3ca79fd71c1b538c813 WatchSource:0}: Error finding container 160fb5c6367c72ba36757b0cb3591e0581b568e5814ca3ca79fd71c1b538c813: Status 404 returned error can't find the container with id 160fb5c6367c72ba36757b0cb3591e0581b568e5814ca3ca79fd71c1b538c813 Nov 02 15:16:28.210338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.210264 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb"] Nov 02 15:16:28.216394 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:28.216367 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2d82f4_e05a_4737_970c_38a126c560b8.slice/crio-5359c501c13732d499827aa1a03687488e97fe2e91e98b9abdea45ca3bf86b7b WatchSource:0}: Error finding container 5359c501c13732d499827aa1a03687488e97fe2e91e98b9abdea45ca3bf86b7b: Status 404 returned error can't find the container with id 5359c501c13732d499827aa1a03687488e97fe2e91e98b9abdea45ca3bf86b7b Nov 02 15:16:28.862057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.862014 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" event={"ID":"0a2d82f4-e05a-4737-970c-38a126c560b8","Type":"ContainerStarted","Data":"cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770"} Nov 02 15:16:28.862569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.862068 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" event={"ID":"0a2d82f4-e05a-4737-970c-38a126c560b8","Type":"ContainerStarted","Data":"5359c501c13732d499827aa1a03687488e97fe2e91e98b9abdea45ca3bf86b7b"} Nov 02 15:16:28.863709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.863679 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" event={"ID":"c08d279e-fe25-49da-ab6c-8d83c0894b8e","Type":"ContainerStarted","Data":"57fb3230445f3aeff806844e1756f7c4c1f9056323d8786fe02ffe024b3a107e"} Nov 02 15:16:28.863817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.863724 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" event={"ID":"c08d279e-fe25-49da-ab6c-8d83c0894b8e","Type":"ContainerStarted","Data":"160fb5c6367c72ba36757b0cb3591e0581b568e5814ca3ca79fd71c1b538c813"} Nov 02 15:16:28.976224 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:28.976154 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podStartSLOduration=1.97613627 podStartE2EDuration="1.97613627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:28.975707622 +0000 UTC m=+1143.140260162" watchObservedRunningTime="2026-11-02 15:16:28.97613627 +0000 UTC m=+1143.140688810" Nov 02 15:16:29.802269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:29.802234 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:29.870983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:29.870945 5091 generic.go:356] "Generic (PLEG): container finished" podID="c08d279e-fe25-49da-ab6c-8d83c0894b8e" containerID="57fb3230445f3aeff806844e1756f7c4c1f9056323d8786fe02ffe024b3a107e" exitCode=0 Nov 02 15:16:29.871385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:29.871039 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" event={"ID":"c08d279e-fe25-49da-ab6c-8d83c0894b8e","Type":"ContainerDied","Data":"57fb3230445f3aeff806844e1756f7c4c1f9056323d8786fe02ffe024b3a107e"} Nov 02 15:16:31.010977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.010952 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:31.138391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.138305 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj\" (UniqueName: \"kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj\") pod \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " Nov 02 15:16:31.138598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.138404 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca\") pod \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " Nov 02 15:16:31.138598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.138467 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca\") pod \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\" (UID: \"c08d279e-fe25-49da-ab6c-8d83c0894b8e\") " Nov 02 15:16:31.138825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.138796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca" pod "c08d279e-fe25-49da-ab6c-8d83c0894b8e" (UID: "c08d279e-fe25-49da-ab6c-8d83c0894b8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:31.140696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.140664 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca" pod "c08d279e-fe25-49da-ab6c-8d83c0894b8e" (UID: "c08d279e-fe25-49da-ab6c-8d83c0894b8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:31.140696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.140676 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj" pod "c08d279e-fe25-49da-ab6c-8d83c0894b8e" (UID: "c08d279e-fe25-49da-ab6c-8d83c0894b8e"). InnerVolumeSpecName "kube-api-access-pmhqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:31.239804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.239762 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:31.239804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.239798 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c08d279e-fe25-49da-ab6c-8d83c0894b8e-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:31.239804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.239809 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhqj\" (UniqueName: \"kubernetes.io/projected/c08d279e-fe25-49da-ab6c-8d83c0894b8e-kube-api-access-pmhqj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:31.883896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.883857 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" event={"ID":"c08d279e-fe25-49da-ab6c-8d83c0894b8e","Type":"ContainerDied","Data":"160fb5c6367c72ba36757b0cb3591e0581b568e5814ca3ca79fd71c1b538c813"} Nov 02 15:16:31.884086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.883922 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="160fb5c6367c72ba36757b0cb3591e0581b568e5814ca3ca79fd71c1b538c813" Nov 02 15:16:31.884086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:31.883878 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6" Nov 02 15:16:37.627297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.627257 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m"] Nov 02 15:16:37.628051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.628024 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c08d279e-fe25-49da-ab6c-8d83c0894b8e" containerName="revoke-keys" Nov 02 15:16:37.628124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.628052 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08d279e-fe25-49da-ab6c-8d83c0894b8e" containerName="revoke-keys" Nov 02 15:16:37.628262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.628249 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08d279e-fe25-49da-ab6c-8d83c0894b8e" containerName="revoke-keys" Nov 02 15:16:37.632213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.632188 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.704653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.704608 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.704835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.704677 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c24t\" (UniqueName: \"kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.704873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.704832 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.728890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.728856 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:37.728890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.728867 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:37.731796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.731769 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m"] Nov 02 15:16:37.803078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.803049 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:37.805386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.805361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.805543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.805393 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.805543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.805434 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c24t\" (UniqueName: \"kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.806211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.806180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.807761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.807737 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.820245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.820218 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:37.864054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.864017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c24t\" (UniqueName: \"kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t\") pod \"maas-api-revoke-sub-271a0cc5-26d8m\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:37.930451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.930358 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:16:37.946482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:37.946450 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:38.191930 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.191849 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8"] Nov 02 15:16:38.196849 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.196829 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.213493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.213469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jbj\" (UniqueName: \"kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.213632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.213528 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.213632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.213577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.256292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.256254 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8"] Nov 02 15:16:38.314701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.314663 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jbj\" (UniqueName: \"kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.314897 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.314724 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.314897 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.314756 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.315594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.315558 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.317135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.317107 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.321583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.321555 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m"] Nov 02 15:16:38.326655 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:38.326631 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930382b7_1e4a_44d7_b4b0_51cab6e94e10.slice/crio-dfd1f63acce32b716244b39b9125534af698249dbe7b4c3990db571dbb6cec63 WatchSource:0}: Error finding container dfd1f63acce32b716244b39b9125534af698249dbe7b4c3990db571dbb6cec63: Status 404 returned error can't find the container with id dfd1f63acce32b716244b39b9125534af698249dbe7b4c3990db571dbb6cec63 Nov 02 15:16:38.374111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.374084 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jbj\" (UniqueName: \"kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj\") pod \"maas-api-revoke-sub-c67ea374-rn4p8\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.511692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.511652 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:38.927144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.927089 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" event={"ID":"930382b7-1e4a-44d7-b4b0-51cab6e94e10","Type":"ContainerStarted","Data":"49983950a47ba0dd722649a1dfb2134460cbf4dd473ec921de628a5affb6d57c"} Nov 02 15:16:38.927630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.927165 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" event={"ID":"930382b7-1e4a-44d7-b4b0-51cab6e94e10","Type":"ContainerStarted","Data":"dfd1f63acce32b716244b39b9125534af698249dbe7b4c3990db571dbb6cec63"} Nov 02 15:16:38.955800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:38.955772 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8"] Nov 02 15:16:38.961994 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:38.961959 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0132b239_fec0_426a_b436_db370fd947fd.slice/crio-41af275032e6e4278e090e7376cf89f3ebbdf61c3d4c231b52396aa0c792c23f WatchSource:0}: Error finding container 41af275032e6e4278e090e7376cf89f3ebbdf61c3d4c231b52396aa0c792c23f: Status 404 returned error can't find the container with id 41af275032e6e4278e090e7376cf89f3ebbdf61c3d4c231b52396aa0c792c23f Nov 02 15:16:39.040430 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:39.040369 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" podStartSLOduration=2.040353307 podStartE2EDuration="2.040353307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:39.038307882 +0000 UTC m=+1153.202860423" watchObservedRunningTime="2026-11-02 15:16:39.040353307 +0000 UTC m=+1153.204905843" Nov 02 15:16:39.936779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:39.936736 5091 generic.go:356] "Generic (PLEG): container finished" podID="0132b239-fec0-426a-b436-db370fd947fd" containerID="c0c62f63b89a5d34f41e9f5bb7cc3c3f8ca7f41895ecd72693ab49f59ab0a479" exitCode=0 Nov 02 15:16:39.937223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:39.936816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" event={"ID":"0132b239-fec0-426a-b436-db370fd947fd","Type":"ContainerDied","Data":"c0c62f63b89a5d34f41e9f5bb7cc3c3f8ca7f41895ecd72693ab49f59ab0a479"} Nov 02 15:16:39.937223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:39.936881 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" event={"ID":"0132b239-fec0-426a-b436-db370fd947fd","Type":"ContainerStarted","Data":"41af275032e6e4278e090e7376cf89f3ebbdf61c3d4c231b52396aa0c792c23f"} Nov 02 15:16:39.938567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:39.938545 5091 generic.go:356] "Generic (PLEG): container finished" podID="930382b7-1e4a-44d7-b4b0-51cab6e94e10" containerID="49983950a47ba0dd722649a1dfb2134460cbf4dd473ec921de628a5affb6d57c" exitCode=0 Nov 02 15:16:39.938696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:39.938578 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" event={"ID":"930382b7-1e4a-44d7-b4b0-51cab6e94e10","Type":"ContainerDied","Data":"49983950a47ba0dd722649a1dfb2134460cbf4dd473ec921de628a5affb6d57c"} Nov 02 15:16:40.289196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.289154 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg"] Nov 02 15:16:40.293558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.293501 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338171 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338222 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338300 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25ddq\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338356 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338400 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338464 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.338676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.338604 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.343936 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.343910 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fce776-openshift-default-dockercfg-5zhzc\"" Nov 02 15:16:40.352049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.352022 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg"] Nov 02 15:16:40.439289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439250 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439535 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439593 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25ddq\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439624 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439697 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439820 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.439992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.439849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.440074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.440053 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.440146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.440116 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.440146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.440137 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.442078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.442058 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.442163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.442110 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.498217 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.498173 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.498731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.498701 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25ddq\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:40.606726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:40.606642 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:41.038366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.038330 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg"] Nov 02 15:16:41.047919 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.047868 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:41.048084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.047972 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:41.048084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.048024 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:41.113339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.113308 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:41.132824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.132791 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:41.147424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.147384 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca\") pod \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " Nov 02 15:16:41.147621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.147450 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t\" (UniqueName: \"kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t\") pod \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " Nov 02 15:16:41.147707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.147689 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca\") pod \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\" (UID: \"930382b7-1e4a-44d7-b4b0-51cab6e94e10\") " Nov 02 15:16:41.148050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.148024 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca" pod "930382b7-1e4a-44d7-b4b0-51cab6e94e10" (UID: "930382b7-1e4a-44d7-b4b0-51cab6e94e10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:41.150197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.150163 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t" pod "930382b7-1e4a-44d7-b4b0-51cab6e94e10" (UID: "930382b7-1e4a-44d7-b4b0-51cab6e94e10"). InnerVolumeSpecName "kube-api-access-7c24t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:41.150306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.150259 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca" pod "930382b7-1e4a-44d7-b4b0-51cab6e94e10" (UID: "930382b7-1e4a-44d7-b4b0-51cab6e94e10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:41.248429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.248320 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca\") pod \"0132b239-fec0-426a-b436-db370fd947fd\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " Nov 02 15:16:41.248429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.248399 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca\") pod \"0132b239-fec0-426a-b436-db370fd947fd\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " Nov 02 15:16:41.248705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.248536 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj\" (UniqueName: \"kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj\") pod \"0132b239-fec0-426a-b436-db370fd947fd\" (UID: \"0132b239-fec0-426a-b436-db370fd947fd\") " Nov 02 15:16:41.248705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.248684 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca" pod "0132b239-fec0-426a-b436-db370fd947fd" (UID: "0132b239-fec0-426a-b436-db370fd947fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:41.249246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.249228 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:41.249246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.249246 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c24t\" (UniqueName: \"kubernetes.io/projected/930382b7-1e4a-44d7-b4b0-51cab6e94e10-kube-api-access-7c24t\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:41.249405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.249257 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930382b7-1e4a-44d7-b4b0-51cab6e94e10-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:41.249405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.249267 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0132b239-fec0-426a-b436-db370fd947fd-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:41.250572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.250543 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca" pod "0132b239-fec0-426a-b436-db370fd947fd" (UID: "0132b239-fec0-426a-b436-db370fd947fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:41.250663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.250628 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj" pod "0132b239-fec0-426a-b436-db370fd947fd" (UID: "0132b239-fec0-426a-b436-db370fd947fd"). InnerVolumeSpecName "kube-api-access-q6jbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:41.350717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.350682 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jbj\" (UniqueName: \"kubernetes.io/projected/0132b239-fec0-426a-b436-db370fd947fd-kube-api-access-q6jbj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:41.350717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.350718 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0132b239-fec0-426a-b436-db370fd947fd-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:41.959857 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.959827 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" Nov 02 15:16:41.961546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.961503 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m" event={"ID":"930382b7-1e4a-44d7-b4b0-51cab6e94e10","Type":"ContainerDied","Data":"dfd1f63acce32b716244b39b9125534af698249dbe7b4c3990db571dbb6cec63"} Nov 02 15:16:41.961707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.961565 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd1f63acce32b716244b39b9125534af698249dbe7b4c3990db571dbb6cec63" Nov 02 15:16:41.961707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.961581 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" event={"ID":"5fe428c1-d9b9-4c39-9232-6faa98e597d7","Type":"ContainerStarted","Data":"d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226"} Nov 02 15:16:41.961707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.961604 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" event={"ID":"5fe428c1-d9b9-4c39-9232-6faa98e597d7","Type":"ContainerStarted","Data":"98030b5b9f4f5db969a982996f5d6ea63c8adcb3ec952b973668d7eb0cf3aa89"} Nov 02 15:16:41.964998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.964976 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" event={"ID":"0132b239-fec0-426a-b436-db370fd947fd","Type":"ContainerDied","Data":"41af275032e6e4278e090e7376cf89f3ebbdf61c3d4c231b52396aa0c792c23f"} Nov 02 15:16:41.964998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.964989 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8" Nov 02 15:16:41.965171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:41.965011 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41af275032e6e4278e090e7376cf89f3ebbdf61c3d4c231b52396aa0c792c23f" Nov 02 15:16:42.146539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:42.146370 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podStartSLOduration=2.146348265 podStartE2EDuration="2.146348265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:42.124560793 +0000 UTC m=+1156.289113330" watchObservedRunningTime="2026-11-02 15:16:42.146348265 +0000 UTC m=+1156.310900810" Nov 02 15:16:42.607337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:42.607305 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:42.608791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:42.608767 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:42.608902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:42.608812 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 15:16:43.607874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:43.607836 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:43.608241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:43.607884 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 15:16:44.607830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:44.607791 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:44.608015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:44.607842 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 15:16:45.608015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:45.607978 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:45.608403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:45.608027 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 15:16:46.607100 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:46.607060 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 02 15:16:46.607317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:46.607134 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 02 15:16:47.960992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:47.960954 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:47.961474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:47.961338 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:47.962508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:47.962487 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:16:51.498126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.498087 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8"] Nov 02 15:16:51.498881 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.498861 5091 cpu_manager.go:414] "containerMap: removing container" podUID="930382b7-1e4a-44d7-b4b0-51cab6e94e10" containerName="revoke-keys" Nov 02 15:16:51.498881 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.498881 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930382b7-1e4a-44d7-b4b0-51cab6e94e10" containerName="revoke-keys" Nov 02 15:16:51.499023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.498902 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0132b239-fec0-426a-b436-db370fd947fd" containerName="revoke-keys" Nov 02 15:16:51.499023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.498907 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0132b239-fec0-426a-b436-db370fd947fd" containerName="revoke-keys" Nov 02 15:16:51.499023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.499014 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="930382b7-1e4a-44d7-b4b0-51cab6e94e10" containerName="revoke-keys" Nov 02 15:16:51.499023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.499024 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0132b239-fec0-426a-b436-db370fd947fd" containerName="revoke-keys" Nov 02 15:16:51.502941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.502918 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569258 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569303 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569335 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569472 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569550 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569644 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569710 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569742 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhk9\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.569862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.569848 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.605905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.605864 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8"] Nov 02 15:16:51.671195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671226 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671263 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671298 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671331 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671483 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671512 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhk9\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671720 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671779 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.671967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671929 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.672008 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.671985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.672458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.672427 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.673773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.673745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.674065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.674046 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.737082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.737048 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.737287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.737264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhk9\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-z59f8\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:51.818200 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:51.818112 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:52.257384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:52.257347 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8"] Nov 02 15:16:52.263185 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:52.263145 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbc2167_c19a_43a3_be40_61607f5addf3.slice/crio-5616ef04ac58807c424470ba88fd3556e2b7109891265d9c4bc7868deccb21fe WatchSource:0}: Error finding container 5616ef04ac58807c424470ba88fd3556e2b7109891265d9c4bc7868deccb21fe: Status 404 returned error can't find the container with id 5616ef04ac58807c424470ba88fd3556e2b7109891265d9c4bc7868deccb21fe Nov 02 15:16:52.265785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:52.265752 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:52.265861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:52.265822 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:52.265861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:52.265851 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:53.055095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:53.055058 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" event={"ID":"5dbc2167-c19a-43a3-be40-61607f5addf3","Type":"ContainerStarted","Data":"61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef"} Nov 02 15:16:53.055690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:53.055137 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" event={"ID":"5dbc2167-c19a-43a3-be40-61607f5addf3","Type":"ContainerStarted","Data":"5616ef04ac58807c424470ba88fd3556e2b7109891265d9c4bc7868deccb21fe"} Nov 02 15:16:53.221951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:53.221887 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podStartSLOduration=2.22186327 podStartE2EDuration="2.22186327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:53.218248839 +0000 UTC m=+1167.382801381" watchObservedRunningTime="2026-11-02 15:16:53.22186327 +0000 UTC m=+1167.386415871" Nov 02 15:16:53.818322 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:53.818279 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:16:53.820051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:53.820024 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:53.820186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:53.820073 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:54.254151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.253491 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4"] Nov 02 15:16:54.258410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.258386 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.302633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.302578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46tt5\" (UniqueName: \"kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.302633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.302623 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.302913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.302767 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.304866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.304835 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4"] Nov 02 15:16:54.308462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.308412 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:16:54.308884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.308849 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:16:54.404243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.404202 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46tt5\" (UniqueName: \"kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.404243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.404246 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.404465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.404307 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.405176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.405152 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.406817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.406797 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.463587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.463546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46tt5\" (UniqueName: \"kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5\") pod \"maas-api-revoke-sub-6301bc7e-62pw4\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.573159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.573051 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:54.819179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.819145 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:54.819355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.819205 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:54.974351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:54.974314 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4"] Nov 02 15:16:54.979799 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:54.979768 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af7657c_bf64_424d_80fe_4a681a80707f.slice/crio-19ae7844d01256e8ed8cd808ef6e13dbd5d84a4c429c669725854d24848b0c67 WatchSource:0}: Error finding container 19ae7844d01256e8ed8cd808ef6e13dbd5d84a4c429c669725854d24848b0c67: Status 404 returned error can't find the container with id 19ae7844d01256e8ed8cd808ef6e13dbd5d84a4c429c669725854d24848b0c67 Nov 02 15:16:55.068478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.068442 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" event={"ID":"5af7657c-bf64-424d-80fe-4a681a80707f","Type":"ContainerStarted","Data":"19ae7844d01256e8ed8cd808ef6e13dbd5d84a4c429c669725854d24848b0c67"} Nov 02 15:16:55.138133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.138091 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t"] Nov 02 15:16:55.169889 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.169801 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.189542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.189431 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t"] Nov 02 15:16:55.214030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.213980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214112 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214194 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214368 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214291 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214368 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214333 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214380 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214496 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214550 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.214644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.214627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hvt8\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.257964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.257935 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-e98976-openshift-default-dockercfg-z9ssr\"" Nov 02 15:16:55.316253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316206 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316287 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316423 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hvt8\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316493 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316584 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316656 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316699 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316721 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316758 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.316783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316780 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.317105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.316941 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.317440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.317419 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.318883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.318863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.319026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.319009 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.377663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.377624 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hvt8\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.377827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.377631 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.483959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.483911 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:55.800359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.800323 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld"] Nov 02 15:16:55.805807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.805782 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.818813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.818788 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:55.818967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.818947 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:55.878980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.878947 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8a2e6f-openshift-default-dockercfg-sqgd8\"" Nov 02 15:16:55.880001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.879976 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld"] Nov 02 15:16:55.924042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.923958 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924004 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924118 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924285 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75x7\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:55.924458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:55.924357 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.025531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.025495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75x7\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.025724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.025571 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.025844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.025822 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.025895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.025878 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.025946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.025934 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.025957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026018 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026043 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026093 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026221 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026204 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026584 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.026788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026769 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.032887 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.026984 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.037538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.033792 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.037538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.034373 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.058535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.054743 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t"] Nov 02 15:16:56.070460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.070417 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.070597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.070553 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.070665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.070598 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.078313 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.078288 5091 generic.go:356] "Generic (PLEG): container finished" podID="5af7657c-bf64-424d-80fe-4a681a80707f" containerID="a8767940fdc18c7dfb77340a69164ecf1e5861022efaba9af06695a72a2c2a6b" exitCode=0 Nov 02 15:16:56.078423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.078367 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" event={"ID":"5af7657c-bf64-424d-80fe-4a681a80707f","Type":"ContainerDied","Data":"a8767940fdc18c7dfb77340a69164ecf1e5861022efaba9af06695a72a2c2a6b"} Nov 02 15:16:56.079563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.079541 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" event={"ID":"e1391062-b3d8-4c72-882b-c1d4b765275d","Type":"ContainerStarted","Data":"5ba6c9c3e7715b786d25a1c8ca38a1d2890cbed795aa45fb51e4bdabda8b51a0"} Nov 02 15:16:56.087285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.087264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75x7\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.088491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.088464 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.124318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.124136 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:56.281552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.281473 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq"] Nov 02 15:16:56.286748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.286724 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.366224 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.366189 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq"] Nov 02 15:16:56.398048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.397991 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-71d1eb-openshift-default-dockercfg-zn6lk\"" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.464820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.464877 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qw8\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.464902 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.464937 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.464959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.464996 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.465015 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.465043 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.466645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.465073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.559255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.559227 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld"] Nov 02 15:16:56.562948 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:16:56.562918 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aa62505_cde6_4da5_821a_93400bef0cfd.slice/crio-04640d5cc45c2ffd5f68fcfdea458ee22a3e8b696ee75240a745253fc30a81c3 WatchSource:0}: Error finding container 04640d5cc45c2ffd5f68fcfdea458ee22a3e8b696ee75240a745253fc30a81c3: Status 404 returned error can't find the container with id 04640d5cc45c2ffd5f68fcfdea458ee22a3e8b696ee75240a745253fc30a81c3 Nov 02 15:16:56.566189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.566151 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.566266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.566242 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.566329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.566282 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:56.566450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.566408 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qw8\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.566951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.566909 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.566474 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567299 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567325 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567446 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567379 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567446 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567401 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.567586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.568550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.567806 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.568550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.568242 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.568550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.568377 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.571741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.569740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.571741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.570745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.571741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.571217 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.604207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.604135 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk"] Nov 02 15:16:56.609626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.609602 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.661541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.646620 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qw8\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.661541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.647022 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.661541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.661215 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-891723-openshift-default-dockercfg-c9wsk\"" Nov 02 15:16:56.680883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.680752 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk"] Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771316 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771484 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771560 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771612 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771655 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlkp\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.774617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.771813 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.819826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.819315 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:56.819826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.819376 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885261 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlkp\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885412 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.885544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885538 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.886159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.885588 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.889471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.886387 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.889471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.886894 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.889471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.887683 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.889471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.888552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.892167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.891915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.892167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.892126 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.895014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.894992 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.905772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.905648 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:56.969426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.968091 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.969426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.969132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlkp\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:56.993543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:56.977235 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:16:57.119541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.117935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" event={"ID":"e1391062-b3d8-4c72-882b-c1d4b765275d","Type":"ContainerStarted","Data":"5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73"} Nov 02 15:16:57.123563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.122085 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" event={"ID":"0aa62505-cde6-4da5-821a-93400bef0cfd","Type":"ContainerStarted","Data":"9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c"} Nov 02 15:16:57.123563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.122130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" event={"ID":"0aa62505-cde6-4da5-821a-93400bef0cfd","Type":"ContainerStarted","Data":"04640d5cc45c2ffd5f68fcfdea458ee22a3e8b696ee75240a745253fc30a81c3"} Nov 02 15:16:57.124851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.124809 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:16:57.207820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.207463 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.207820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.207540 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:16:57.415623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.414108 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podStartSLOduration=2.414092737 podStartE2EDuration="2.414092737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:57.312734176 +0000 UTC m=+1171.477286711" watchObservedRunningTime="2026-11-02 15:16:57.414092737 +0000 UTC m=+1171.578645277" Nov 02 15:16:57.417235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.416873 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq"] Nov 02 15:16:57.500579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.495877 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.500579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.495976 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.500579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.496018 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:57.504482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.502787 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:16:57.506737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.506646 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.506737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.506714 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:16:57.519154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.519085 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podStartSLOduration=2.519062608 podStartE2EDuration="2.519062608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:57.497346264 +0000 UTC m=+1171.661898807" watchObservedRunningTime="2026-11-02 15:16:57.519062608 +0000 UTC m=+1171.683615148" Nov 02 15:16:57.712957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.712915 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk"] Nov 02 15:16:57.862169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.862130 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:57.865671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:57.865623 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:58.272142 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.202676 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.272142 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.202821 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:16:58.300972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.300911 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:58.301329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.301292 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:58.318743 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.318682 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:16:58.417993 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:16:58.417950 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:16:58.502790 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.468450 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" event={"ID":"6b96e8e4-61a6-4f54-8021-772d390f04ec","Type":"ContainerStarted","Data":"cc2032774108827bab24ded0c134222de61501cc481af744509f144ca8a59d7b"} Nov 02 15:16:58.522085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.520860 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" event={"ID":"ab86c317-ec5b-4378-b124-3bc39162157d","Type":"ContainerStarted","Data":"1fcae796017b91b542398e58c015c32b6fc4730fbed15c49674a655c9daa4241"} Nov 02 15:16:58.522085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.493307 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.522085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.520937 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" event={"ID":"5af7657c-bf64-424d-80fe-4a681a80707f","Type":"ContainerDied","Data":"19ae7844d01256e8ed8cd808ef6e13dbd5d84a4c429c669725854d24848b0c67"} Nov 02 15:16:58.522085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.520963 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:16:58.522085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.520967 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19ae7844d01256e8ed8cd808ef6e13dbd5d84a4c429c669725854d24848b0c67" Nov 02 15:16:58.522085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.515794 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:58.650558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.647909 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5\" (UniqueName: \"kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5\") pod \"5af7657c-bf64-424d-80fe-4a681a80707f\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " Nov 02 15:16:58.650558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.647965 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca\") pod \"5af7657c-bf64-424d-80fe-4a681a80707f\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " Nov 02 15:16:58.650558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.648044 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca\") pod \"5af7657c-bf64-424d-80fe-4a681a80707f\" (UID: \"5af7657c-bf64-424d-80fe-4a681a80707f\") " Nov 02 15:16:58.650558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.649926 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podStartSLOduration=2.649908697 podStartE2EDuration="2.649908697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:58.64813594 +0000 UTC m=+1172.812688481" watchObservedRunningTime="2026-11-02 15:16:58.649908697 +0000 UTC m=+1172.814461238" Nov 02 15:16:58.653365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.653326 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca" pod "5af7657c-bf64-424d-80fe-4a681a80707f" (UID: "5af7657c-bf64-424d-80fe-4a681a80707f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:16:58.664191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.664161 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5" pod "5af7657c-bf64-424d-80fe-4a681a80707f" (UID: "5af7657c-bf64-424d-80fe-4a681a80707f"). InnerVolumeSpecName "kube-api-access-46tt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:16:58.698550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.696091 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca" pod "5af7657c-bf64-424d-80fe-4a681a80707f" (UID: "5af7657c-bf64-424d-80fe-4a681a80707f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:16:58.749745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.749707 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46tt5\" (UniqueName: \"kubernetes.io/projected/5af7657c-bf64-424d-80fe-4a681a80707f-kube-api-access-46tt5\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:58.749745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.749747 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:58.749969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.749764 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5af7657c-bf64-424d-80fe-4a681a80707f-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:16:58.823098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.820168 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:58.823098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.820227 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:58.914545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:58.906991 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:16:59.124863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.124830 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.125123 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.125093 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:16:59.502598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.498646 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.502598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.498709 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:16:59.600294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.599057 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" event={"ID":"ab86c317-ec5b-4378-b124-3bc39162157d","Type":"ContainerStarted","Data":"6fe38af1bf49e510cff9ce2bdf4545c98894a605fdaf2d44364e729a55ff8a2e"} Nov 02 15:16:59.609409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.609375 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" event={"ID":"6b96e8e4-61a6-4f54-8021-772d390f04ec","Type":"ContainerStarted","Data":"17d110cd7f69f7f116896702f92b936e968b15e575e2d2444a29d51f17f2481b"} Nov 02 15:16:59.609735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.609714 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4" Nov 02 15:16:59.737528 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.729478 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podStartSLOduration=3.72945749 podStartE2EDuration="3.72945749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:16:59.728085545 +0000 UTC m=+1173.892638087" watchObservedRunningTime="2026-11-02 15:16:59.72945749 +0000 UTC m=+1173.894010031" Nov 02 15:16:59.822061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.821956 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.822292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.822258 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:16:59.909488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.909352 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:16:59.909488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.909414 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:16:59.989540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:16:59.989255 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:17:00.005542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.004195 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.005542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.004253 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:00.025813 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:17:00.025775 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af7657c_bf64_424d_80fe_4a681a80707f.slice\": RecentStats: unable to find data in memory cache]" Nov 02 15:17:00.126556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.125694 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.126556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.125771 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:17:00.487158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.487051 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.487440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.487395 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:17:00.583678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.583631 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:00.819062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.818963 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.819316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.819282 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 02 15:17:00.921793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.921757 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.922036 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.922004 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:00.925619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.925584 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:00.980410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.980368 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:00.980736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:00.980705 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:01.154542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.129647 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.154542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.129725 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:17:01.374539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.373604 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.452598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.451629 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.510543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.502641 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.510543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.502706 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:17:01.593539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.592612 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.593539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.593095 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:17:01.637543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.636625 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:17:01.894538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.887611 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.902540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.896318 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:17:01.916557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.910152 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:01.916557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.910209 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:01.943559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.941618 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:01.943559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:01.942087 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:17:02.117706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.117663 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.118002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.117972 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:02.127652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.127603 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.329608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.257901 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.329608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.257995 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:17:02.382572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.382531 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.383239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.383214 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:17:02.467768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.467721 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.468441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.468416 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:17:02.493030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.492992 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.495048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.495014 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:17:02.599598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.596307 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.677543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.674386 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.800768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.800734 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:17:02.898608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.887617 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-z59f8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:17:02.898608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.887690 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:02.898608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.890374 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:17:02.898608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.890759 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:17:02.898608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.893288 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:17:02.953540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.909719 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:02.953863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:02.953827 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:03.078562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.075927 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:03.078562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.075985 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:03.145538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.120821 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:17:03.147062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.147037 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:17:03.147646 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.147579 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:03.147809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.147790 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:17:03.170593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.146356 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.170593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.154736 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:17:03.172282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.172257 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:17:03.423934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.423809 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:17:03.573543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.510813 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:17:03.574081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.530576 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 02 15:17:03.574268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.574236 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 02 15:17:03.574396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.535646 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:17:03.574586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.574562 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.611540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.602612 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:17:03.809592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.806791 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.809592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.807556 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.907540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.901133 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:03.919077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.912361 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:03.919077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.912419 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:03.981546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.977870 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:03.989654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:03.989613 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:04.155905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.155867 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:04.180664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.180606 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:17:04.180906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.160589 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.427708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.427604 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.428004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.427976 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:17:04.527654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.527602 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:17:04.577667 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.577610 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.599574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.599539 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:17:04.617080 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.617040 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.822600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.821686 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.939536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.939074 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.939536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.939498 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:17:04.979556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.972780 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:17:04.984754 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.984694 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:04.985408 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.985385 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:17:04.999980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:04.999954 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:17:05.013807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.013769 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:05.014070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.014040 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:05.125733 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.125698 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:05.126023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.125989 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:05.133547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.126455 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 02 15:17:05.133806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.133771 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 02 15:17:05.219807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.219756 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.429621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.422000 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:17:05.526715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.526672 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:17:05.526999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.526965 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:05.592981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.592951 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:17:05.593918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:05.593880 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.044956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.042718 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:06.044956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.042781 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:06.044956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.044853 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:06.044956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.044896 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:06.494274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.494222 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Nov 02 15:17:06.550645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.495602 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.576640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.576596 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.608230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.608187 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.609106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.609074 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.614599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.611048 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:17:06.722551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.722495 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:17:06.722876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.722857 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:17:06.723776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.723744 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:17:06.725630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.725614 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:17:06.742507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.742482 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:17:06.794556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.794459 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:17:06.794917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.794891 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:17:06.795700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.795681 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:17:06.906932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.906902 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 02 15:17:06.907106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.906952 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 02 15:17:06.978083 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.978050 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:06.978229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:06.978107 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:07.978251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:07.978211 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 02 15:17:07.978703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:07.978276 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 02 15:17:08.645248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:08.645210 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:17:08.645647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:08.645623 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:17:08.646690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:08.646665 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:17:09.327306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:09.327272 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:17:09.327846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:09.327826 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:17:09.329065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:09.329041 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:17:26.097111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:26.097076 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:17:26.101548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:26.101500 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:17:27.934208 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:17:27.934174 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:17:51.144219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.144174 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5"] Nov 02 15:17:51.144998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.144974 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5af7657c-bf64-424d-80fe-4a681a80707f" containerName="revoke-keys" Nov 02 15:17:51.145092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.145000 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af7657c-bf64-424d-80fe-4a681a80707f" containerName="revoke-keys" Nov 02 15:17:51.145175 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.145163 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af7657c-bf64-424d-80fe-4a681a80707f" containerName="revoke-keys" Nov 02 15:17:51.149075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.149052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.201596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.201566 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5"] Nov 02 15:17:51.245138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.245113 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fce776\"/\"distinct-w2-fce776-kserve-self-signed-certs\"" Nov 02 15:17:51.319753 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.319717 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4sj4\" (UniqueName: \"kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.319950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.319822 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.319950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.319877 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.319950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.319923 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.320149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.320009 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.320149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.320131 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.420961 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4sj4\" (UniqueName: \"kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421018 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421059 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421093 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421111 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421170 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421576 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421592 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.421714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.421651 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.423655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.423633 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.424106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.424085 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.480457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.480427 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4sj4\" (UniqueName: \"kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4\") pod \"distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:51.764706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:51.764619 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:52.140495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:52.140440 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5"] Nov 02 15:17:52.144789 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:17:52.144765 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc688f54_c7e1_4244_9a54_c47dc066520a.slice/crio-e30641e19139463c291f7441f947c31b385126a47a85a87b195d7bc590ad9f7a WatchSource:0}: Error finding container e30641e19139463c291f7441f947c31b385126a47a85a87b195d7bc590ad9f7a: Status 404 returned error can't find the container with id e30641e19139463c291f7441f947c31b385126a47a85a87b195d7bc590ad9f7a Nov 02 15:17:53.046951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:53.042730 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" event={"ID":"fc688f54-c7e1-4244-9a54-c47dc066520a","Type":"ContainerStarted","Data":"006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419"} Nov 02 15:17:53.047299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:53.047261 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" event={"ID":"fc688f54-c7e1-4244-9a54-c47dc066520a","Type":"ContainerStarted","Data":"e30641e19139463c291f7441f947c31b385126a47a85a87b195d7bc590ad9f7a"} Nov 02 15:17:53.161102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:53.161048 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podStartSLOduration=2.161034796 podStartE2EDuration="2.161034796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:17:53.157773982 +0000 UTC m=+1227.322326523" watchObservedRunningTime="2026-11-02 15:17:53.161034796 +0000 UTC m=+1227.325587374" Nov 02 15:17:53.764830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:17:53.764797 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:17:57.934129 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:17:57.934083 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:18:01.765128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:01.765094 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:18:01.778126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:01.778102 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:18:02.113690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:02.113610 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:18:10.748386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.748346 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4"] Nov 02 15:18:10.753103 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.753083 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822211 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822349 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822381 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822441 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822470 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822626 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.822688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.822676 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwfmg\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.841179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.841149 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4"] Nov 02 15:18:10.923697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923664 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.923877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923703 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.923877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923721 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwfmg\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.923877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.923877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923812 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.923877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923835 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.923877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923858 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923892 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.923923 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.924144 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.924218 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.924256 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.924360 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.924679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.924657 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.926089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.926064 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.926398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.926379 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.982672 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.982642 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:10.982791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:10.982700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwfmg\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:11.068272 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:11.068196 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:11.497182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:11.497146 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4"] Nov 02 15:18:11.501617 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:18:11.501587 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4479f9_c787_48df_beb8_2f9283329572.slice/crio-a78ee55380b1e38cfa874b8183c65d09beda02bb8490036a2370508fd8df73e7 WatchSource:0}: Error finding container a78ee55380b1e38cfa874b8183c65d09beda02bb8490036a2370508fd8df73e7: Status 404 returned error can't find the container with id a78ee55380b1e38cfa874b8183c65d09beda02bb8490036a2370508fd8df73e7 Nov 02 15:18:11.503959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:11.503930 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:11.504031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:11.503992 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:11.504031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:11.504023 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:12.171097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:12.171060 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" event={"ID":"9c4479f9-c787-48df-beb8-2f9283329572","Type":"ContainerStarted","Data":"6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad"} Nov 02 15:18:12.171498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:12.171124 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" event={"ID":"9c4479f9-c787-48df-beb8-2f9283329572","Type":"ContainerStarted","Data":"a78ee55380b1e38cfa874b8183c65d09beda02bb8490036a2370508fd8df73e7"} Nov 02 15:18:12.288225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:12.288170 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podStartSLOduration=2.28815379 podStartE2EDuration="2.28815379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:12.286826617 +0000 UTC m=+1246.451379193" watchObservedRunningTime="2026-11-02 15:18:12.28815379 +0000 UTC m=+1246.452706329" Nov 02 15:18:12.906758 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:12.906722 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn"] Nov 02 15:18:12.910662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:12.910632 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:12.960997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:12.960962 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-624908-openshift-default-dockercfg-zvxm2\"" Nov 02 15:18:13.012155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.012124 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn"] Nov 02 15:18:13.064551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064476 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.064753 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064555 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.064816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064729 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.064816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064794 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zd5b\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.064920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.064920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.065028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064924 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.065028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.064956 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.065133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.065067 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.069107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.069078 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:13.070647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.070620 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:13.070778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.070716 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 15:18:13.166673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.166589 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.166673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.166651 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.166673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.166679 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.166956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.166729 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.166956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.166792 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zd5b\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167269 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167310 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167345 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167382 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167606 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.167775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167753 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.168011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.167986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.168262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.168240 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.169235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.169212 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.169448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.169430 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.226080 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.226043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zd5b\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.226509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.226435 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.524492 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.524404 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:13.989349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.989307 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn"] Nov 02 15:18:13.993488 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:18:13.993464 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca67de83_e0b6_4d04_bd24_92d6ee567b05.slice/crio-b52b3e48d83c20e98e866f4878bf4668b09c85a17fdc4765859a752dbe0f9488 WatchSource:0}: Error finding container b52b3e48d83c20e98e866f4878bf4668b09c85a17fdc4765859a752dbe0f9488: Status 404 returned error can't find the container with id b52b3e48d83c20e98e866f4878bf4668b09c85a17fdc4765859a752dbe0f9488 Nov 02 15:18:13.997525 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.997479 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:13.997595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.997566 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:13.997639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:13.997597 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:14.068657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:14.068623 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:14.068833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:14.068679 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 15:18:14.197879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:14.197845 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" event={"ID":"ca67de83-e0b6-4d04-bd24-92d6ee567b05","Type":"ContainerStarted","Data":"a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a"} Nov 02 15:18:14.198067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:14.197902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" event={"ID":"ca67de83-e0b6-4d04-bd24-92d6ee567b05","Type":"ContainerStarted","Data":"b52b3e48d83c20e98e866f4878bf4668b09c85a17fdc4765859a752dbe0f9488"} Nov 02 15:18:14.318241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:14.318137 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podStartSLOduration=2.3181238730000002 podStartE2EDuration="2.318123873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:14.315427915 +0000 UTC m=+1248.479980474" watchObservedRunningTime="2026-11-02 15:18:14.318123873 +0000 UTC m=+1248.482676413" Nov 02 15:18:14.524948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:14.524917 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:15.074539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:15.071913 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:15.074539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:15.071971 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 15:18:15.526095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:15.526053 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:18:15.526600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:15.526147 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 15:18:16.074565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:16.070823 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:16.074565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:16.070881 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 15:18:16.531543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:16.527890 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:18:16.531543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:16.527966 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 15:18:17.072540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:17.071871 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:17.072540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:17.071930 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 15:18:17.525157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:17.525007 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:18:17.525157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:17.525070 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 15:18:18.069076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:18.069036 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 02 15:18:18.069445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:18.069095 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 02 15:18:18.525229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:18.525193 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:18:18.525409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:18.525251 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 15:18:19.074634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:19.074600 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:19.075045 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:19.074982 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:19.075712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:19.075694 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:18:19.525133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:19.525100 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:18:19.525319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:19.525158 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 15:18:20.525028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:20.524986 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 02 15:18:20.525442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:20.525058 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 02 15:18:21.530195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:21.530168 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:21.530697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:21.530624 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:21.531353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:21.531328 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:18:27.935949 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:18:27.935901 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:18:42.078098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.078060 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj"] Nov 02 15:18:42.082310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.082276 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.188678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.188641 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj"] Nov 02 15:18:42.202297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202259 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202406 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202460 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202582 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202615 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202698 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhp7r\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.202907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.202864 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304073 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304212 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304281 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304307 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304399 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304438 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304485 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhp7r\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304551 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.304939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.305050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.304993 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.305219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.305194 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.305287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.305213 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.305385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.305364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.307055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.307031 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.307255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.307235 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.362904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.362832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.363029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.362900 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhp7r\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.396912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.396883 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:42.839973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.839931 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj"] Nov 02 15:18:42.845632 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:18:42.845607 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514400cc_4295_410f_917c_3437fc726399.slice/crio-03aa4485bdff735bb83771a491fa09280e2007be16be197427780fa94da91285 WatchSource:0}: Error finding container 03aa4485bdff735bb83771a491fa09280e2007be16be197427780fa94da91285: Status 404 returned error can't find the container with id 03aa4485bdff735bb83771a491fa09280e2007be16be197427780fa94da91285 Nov 02 15:18:42.848442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.848411 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.848559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.848473 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:42.848559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:42.848500 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:18:43.414186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:43.414151 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" event={"ID":"514400cc-4295-410f-917c-3437fc726399","Type":"ContainerStarted","Data":"a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46"} Nov 02 15:18:43.414637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:43.414211 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" event={"ID":"514400cc-4295-410f-917c-3437fc726399","Type":"ContainerStarted","Data":"03aa4485bdff735bb83771a491fa09280e2007be16be197427780fa94da91285"} Nov 02 15:18:43.540538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:43.540344 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podStartSLOduration=1.5403253700000001 podStartE2EDuration="1.54032537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:18:43.537038519 +0000 UTC m=+1277.701591072" watchObservedRunningTime="2026-11-02 15:18:43.54032537 +0000 UTC m=+1277.704877912" Nov 02 15:18:44.397987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:44.397955 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:44.399488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:44.399461 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 15:18:44.399634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:44.399512 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 15:18:45.397566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:45.397527 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 15:18:45.397968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:45.397600 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 15:18:46.397168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:46.397137 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 15:18:46.397345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:46.397189 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 15:18:47.397675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:47.397642 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 15:18:47.398073 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:47.397693 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 15:18:48.397578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:48.397542 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 15:18:48.397799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:48.397596 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 15:18:49.397383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:49.397346 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 02 15:18:49.397576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:49.397414 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 02 15:18:50.402377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:50.402343 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:50.402919 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:50.402894 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:50.403777 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:18:50.403758 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:18:57.935361 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:18:57.935324 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:19:12.994685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:12.994596 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht"] Nov 02 15:19:13.000248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:12.999943 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.059831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.059801 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-61a835-openshift-default-dockercfg-pq94w\"" Nov 02 15:19:13.104473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104440 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.104674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104497 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.104674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104629 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.104822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.104822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104709 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.104822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104746 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.104822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104816 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swqf7\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.105021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104851 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.105021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.104865 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.112351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.112317 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht"] Nov 02 15:19:13.206232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqf7\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206254 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206277 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206331 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206382 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206424 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206506 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.206894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.206735 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.207028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.207005 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.207097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.207042 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.207097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.207059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.207097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.207062 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.208739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.208712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.208948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.208929 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.264697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.264627 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.264836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.264720 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swqf7\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.323925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.323894 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:13.756443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.756408 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht"] Nov 02 15:19:13.760049 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:19:13.760019 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b1e9b3_1988_458f_af39_7c5d0be754b3.slice/crio-15df99a28773bd489110f1efe6bc1c54c029f9bc2b61a558a98e42af2c9e0fd3 WatchSource:0}: Error finding container 15df99a28773bd489110f1efe6bc1c54c029f9bc2b61a558a98e42af2c9e0fd3: Status 404 returned error can't find the container with id 15df99a28773bd489110f1efe6bc1c54c029f9bc2b61a558a98e42af2c9e0fd3 Nov 02 15:19:13.762367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.762332 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:19:13.762447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.762400 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:19:13.762447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:13.762428 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:19:14.654078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:14.654043 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" event={"ID":"33b1e9b3-1988-458f-af39-7c5d0be754b3","Type":"ContainerStarted","Data":"3484f2b83efe0a69349173ec721a6cf044d52df99afa5dd25b1168af1d4c015f"} Nov 02 15:19:14.655343 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:14.655305 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" event={"ID":"33b1e9b3-1988-458f-af39-7c5d0be754b3","Type":"ContainerStarted","Data":"15df99a28773bd489110f1efe6bc1c54c029f9bc2b61a558a98e42af2c9e0fd3"} Nov 02 15:19:14.868177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:14.868120 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podStartSLOduration=2.868100975 podStartE2EDuration="2.868100975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:14.821059791 +0000 UTC m=+1308.985612332" watchObservedRunningTime="2026-11-02 15:19:14.868100975 +0000 UTC m=+1309.032653519" Nov 02 15:19:15.324707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:15.324672 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:15.326411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:15.326382 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:19:15.326596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:15.326453 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 15:19:16.325273 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:16.325240 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:19:16.325699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:16.325295 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 15:19:17.324668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:17.324624 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:19:17.324874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:17.324680 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 15:19:18.324886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:18.324851 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:19:18.325368 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:18.324924 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 15:19:19.324237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:19.324206 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 02 15:19:19.324389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:19.324270 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 02 15:19:20.676458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:20.676425 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:20.677286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:20.677260 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:20.678546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:20.678510 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:19:27.936853 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:19:27.936816 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:19:43.950789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:43.950752 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws"] Nov 02 15:19:43.954946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:43.954919 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.008478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.008447 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.008665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.008489 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.008665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.008535 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.008665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.008655 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.008813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.008780 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.008869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.008818 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4g5\" (UniqueName: \"kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.055307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.055276 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"kube-root-ca.crt\"" Nov 02 15:19:44.055585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.055566 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"distinct-2-w6-e98976-kserve-self-signed-certs\"" Nov 02 15:19:44.055651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.055586 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"openshift-service-ca.crt\"" Nov 02 15:19:44.055651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.055603 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-e98976\"/\"default-dockercfg-xk8r7\"" Nov 02 15:19:44.060194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.060174 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws"] Nov 02 15:19:44.110007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.109974 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.110377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.110351 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.110507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.110491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.110689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.110672 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.110798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.110774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.110850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.110505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.113617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.111769 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.113760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.113731 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.113896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.113814 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4g5\" (UniqueName: \"kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.120439 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.118338 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.120628 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.120496 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.176650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.176617 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4g5\" (UniqueName: \"kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5\") pod \"distinct-2-w6-e98976-kserve-c48c84d96-n8gws\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.267803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.267708 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:44.666478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.666447 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws"] Nov 02 15:19:44.670815 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:19:44.670788 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda76cfa6b_9055_41e3_a90f_b8f220eb5595.slice/crio-6e34aa8eb9a90a0c1bf545755aa1cea13073a7d897105b0feb75b659b56fc110 WatchSource:0}: Error finding container 6e34aa8eb9a90a0c1bf545755aa1cea13073a7d897105b0feb75b659b56fc110: Status 404 returned error can't find the container with id 6e34aa8eb9a90a0c1bf545755aa1cea13073a7d897105b0feb75b659b56fc110 Nov 02 15:19:44.862161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.862120 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" event={"ID":"a76cfa6b-9055-41e3-a90f-b8f220eb5595","Type":"ContainerStarted","Data":"e8e7d667ea8b37c6415ff672c9f3ae47d95ad15e8f3108f41eed768a0ca569af"} Nov 02 15:19:44.862317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.862174 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" event={"ID":"a76cfa6b-9055-41e3-a90f-b8f220eb5595","Type":"ContainerStarted","Data":"6e34aa8eb9a90a0c1bf545755aa1cea13073a7d897105b0feb75b659b56fc110"} Nov 02 15:19:44.978006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:44.977884 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podStartSLOduration=1.9778664 podStartE2EDuration="1.9778664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:19:44.973707343 +0000 UTC m=+1339.138259882" watchObservedRunningTime="2026-11-02 15:19:44.9778664 +0000 UTC m=+1339.142418942" Nov 02 15:19:45.268125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:45.268047 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:54.268713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:54.268678 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:54.281824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:54.281797 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:54.936285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:19:54.936255 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:19:57.935971 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:19:57.935923 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:20:01.725537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.722029 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn"] Nov 02 15:20:01.727461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.727437 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.780943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.780904 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn"] Nov 02 15:20:01.818984 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.818951 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"kube-root-ca.crt\"" Nov 02 15:20:01.819320 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.819298 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"openshift-service-ca.crt\"" Nov 02 15:20:01.819619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.819599 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"default-dockercfg-86rz9\"" Nov 02 15:20:01.826879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.826842 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.827034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.826994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xztrl\" (UniqueName: \"kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.827131 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.827100 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.827199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.827141 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.827277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.827253 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.827355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.827304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.849249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.849212 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-624908\"/\"distinct-2-w1-624908-kserve-self-signed-certs\"" Nov 02 15:20:01.932543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.932332 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.932543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.932419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.932543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.932525 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.932885 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.932635 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xztrl\" (UniqueName: \"kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.932885 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.932688 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.932885 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.932720 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.934955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.933626 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.934955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.933828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.934955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.934505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.940538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.938935 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:01.946699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:01.946673 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:02.009585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:02.008640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xztrl\" (UniqueName: \"kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl\") pod \"distinct-2-w1-624908-kserve-77b5d689b4-mmwbn\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:02.056920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:02.056877 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:02.585540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:02.585268 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn"] Nov 02 15:20:03.014611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:03.014566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" event={"ID":"f977fedd-4641-483d-8c88-1fd88475c4b4","Type":"ContainerStarted","Data":"426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed"} Nov 02 15:20:03.015245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:03.015217 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" event={"ID":"f977fedd-4641-483d-8c88-1fd88475c4b4","Type":"ContainerStarted","Data":"32f77039790759550d756200cd0515ea9609d3a4e073ed80286daa6cc473b3b2"} Nov 02 15:20:03.061649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:03.061590 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:03.168115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:03.167321 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podStartSLOduration=2.167296554 podStartE2EDuration="2.167296554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:03.130905072 +0000 UTC m=+1357.295457616" watchObservedRunningTime="2026-11-02 15:20:03.167296554 +0000 UTC m=+1357.331849094" Nov 02 15:20:12.058154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:12.058119 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:12.077525 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:12.077495 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:12.090772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:12.090742 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:27.453529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.453491 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94"] Nov 02 15:20:27.457690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.457664 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.499741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.499715 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v"] Nov 02 15:20:27.503871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.503852 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520228 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520200 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520236 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520264 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520433 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520486 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520598 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520673 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmvt\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520711 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520741 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520762 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520793 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.520934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520840 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.520934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6wqn\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.521058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.520936 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.521058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.521016 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.521169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.521055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.547397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.547364 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94"] Nov 02 15:20:27.572329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.572303 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v"] Nov 02 15:20:27.622556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622511 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.622716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622579 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6wqn\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.622716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622613 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.622716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622662 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.622716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622687 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.622716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622706 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.622978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.622978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622932 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.622978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.622946 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623048 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623095 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623133 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623148 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623220 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623250 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmvt\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623318 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623358 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623374 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623388 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623441 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623476 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623542 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623596 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.623821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623683 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.623821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623697 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.624037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623869 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.624037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.623971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.624139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.624035 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.625433 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.625408 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.625664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.625647 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.626006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.625990 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.626223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.626207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.683759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.683729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.692762 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.692725 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.693542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.693500 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6wqn\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.701089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.701068 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmvt\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt\") pod \"e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.776160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.776070 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:27.817224 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:27.817190 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:27.945160 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:20:27.945123 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:20:28.221831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.221597 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v"] Nov 02 15:20:28.225795 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:20:28.224925 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15695c7_0a38_4dd0_9be0_1fae421eca1b.slice/crio-e705fb68093235e87eaf32495a4264cd2dba90d65eae3673e206d0f8475c7156 WatchSource:0}: Error finding container e705fb68093235e87eaf32495a4264cd2dba90d65eae3673e206d0f8475c7156: Status 404 returned error can't find the container with id e705fb68093235e87eaf32495a4264cd2dba90d65eae3673e206d0f8475c7156 Nov 02 15:20:28.225795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.225503 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94"] Nov 02 15:20:28.225795 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:20:28.225786 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4c0a84_d2e4_47cd_a333_1d1cc10b517a.slice/crio-7459c0683526fe81ac0429bdd79707d7d053efc9455c4e7a0bad4d8eba8dd653 WatchSource:0}: Error finding container 7459c0683526fe81ac0429bdd79707d7d053efc9455c4e7a0bad4d8eba8dd653: Status 404 returned error can't find the container with id 7459c0683526fe81ac0429bdd79707d7d053efc9455c4e7a0bad4d8eba8dd653 Nov 02 15:20:28.227847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.227816 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.227959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.227889 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.227959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.227931 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.228998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.228963 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.229139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.229038 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:28.229139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:28.229077 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:29.205968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.205926 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" event={"ID":"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a","Type":"ContainerStarted","Data":"f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e"} Nov 02 15:20:29.206444 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.205991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" event={"ID":"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a","Type":"ContainerStarted","Data":"7459c0683526fe81ac0429bdd79707d7d053efc9455c4e7a0bad4d8eba8dd653"} Nov 02 15:20:29.207983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.207956 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" event={"ID":"c15695c7-0a38-4dd0-9be0-1fae421eca1b","Type":"ContainerStarted","Data":"678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c"} Nov 02 15:20:29.208139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.207996 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" event={"ID":"c15695c7-0a38-4dd0-9be0-1fae421eca1b","Type":"ContainerStarted","Data":"e705fb68093235e87eaf32495a4264cd2dba90d65eae3673e206d0f8475c7156"} Nov 02 15:20:29.594601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.593789 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podStartSLOduration=2.593768699 podStartE2EDuration="2.593768699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:29.460679313 +0000 UTC m=+1383.625231872" watchObservedRunningTime="2026-11-02 15:20:29.593768699 +0000 UTC m=+1383.758321243" Nov 02 15:20:29.778352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.776906 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:29.782702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.782663 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:29.782896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.782725 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:29.818400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.818364 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:29.819977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.819946 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:29.820097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:29.819990 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:30.794574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:30.778968 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:30.794574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:30.779057 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:30.823539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:30.821335 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:30.823539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:30.821490 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:31.533541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.532605 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:31.616540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.615610 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:20:31.780542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.780261 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:31.780542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.780334 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:31.822602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.820112 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:31.822602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.820188 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:31.930540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:31.929662 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.008541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.007931 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:20:32.093605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.092662 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.106538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.106483 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.204035 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.203987 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 15:20:32.431658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.431609 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.432291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.431917 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.597679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.597626 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.598292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.598257 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:20:32.621632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.621574 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.622301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.622278 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:20:32.765592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.760363 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podStartSLOduration=5.760342944 podStartE2EDuration="5.760342944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:29.595241826 +0000 UTC m=+1383.759794379" watchObservedRunningTime="2026-11-02 15:20:32.760342944 +0000 UTC m=+1386.924895485" Nov 02 15:20:32.779541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.772595 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.780098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.780067 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:32.780299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.780271 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:32.857538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.852620 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:32.857538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.852681 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:32.946541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.935162 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:32.946541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:32.936235 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:20:33.019632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.018791 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:20:33.019632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.019267 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:20:33.179550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.178649 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:20:33.214550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.211592 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.214550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.211946 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Nov 02 15:20:33.218542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.215487 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:20:33.218542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.215590 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:20:33.218542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.216641 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.218542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.216959 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:33.463199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.462749 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:20:33.464554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.463536 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:20:33.517539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.514474 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.517539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.514953 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:20:33.559541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.552616 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.594539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.593909 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 02 15:20:33.613580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.605781 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:20:33.613580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.606147 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:20:33.630803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.622639 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:33.865786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.865754 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:33.866352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.866319 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:33.866575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.866555 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:33.886826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.866693 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:33.937315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:33.936834 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.193006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.192906 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:20:34.213555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.212585 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 15:20:34.217391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.217349 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.218396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.218370 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.224710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.224623 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.401084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.387428 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.473463 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.473358 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.532647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.528116 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.608541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.604936 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.631323 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.630553 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.631323 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.630598 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:20:34.631323 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.630414 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:34.633276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.633242 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:20:34.713542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.694744 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:20:34.803434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.803343 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:34.803792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:34.803746 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:35.003687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.003639 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:20:35.074873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.074794 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:20:35.164387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.164337 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.172779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.172739 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:35.193136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.184045 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:20:35.193136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.185347 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:20:35.203324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.202457 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:20:35.223363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.223334 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:20:35.258975 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.258929 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.328328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.328235 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw"] Nov 02 15:20:35.375661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.375614 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c154634-kbbhw"] Nov 02 15:20:35.565737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.565687 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.634598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.615078 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:20:35.634598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.615504 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:20:35.635974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.635930 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.718562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.694852 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:35.810779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.810311 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.810779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.810371 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:35.850558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.848594 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:35.850558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.848645 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:35.872503 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:35.872473 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:20:36.069705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.069605 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:20:36.094222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.094185 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c14cbee7-9491-4dab-98d2-77eca7078839" path="/var/lib/kubelet/pods/c14cbee7-9491-4dab-98d2-77eca7078839/volumes" Nov 02 15:20:36.184110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.184080 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:20:36.218180 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.218156 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:20:36.230173 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.230132 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:20:36.268780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.268748 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:20:36.270889 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.270860 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:20:36.778582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.777179 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 02 15:20:36.778582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.777258 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 02 15:20:36.818351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.818314 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 02 15:20:36.818562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:36.818365 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 02 15:20:37.831394 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:37.831344 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:37.832510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:37.832484 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:37.833611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:37.833584 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:20:37.919126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:37.919086 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:37.920160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:37.920080 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:37.920687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:37.920668 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:20:42.170677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.170640 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74"] Nov 02 15:20:42.177213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.177190 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253253 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253336 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253630 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253698 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq48x\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.253892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.253871 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.254127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.254107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.281553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.281506 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74"] Nov 02 15:20:42.355649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355616 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.355840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355665 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.355840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355692 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.355840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355712 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.355840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355772 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.355840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.355840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq48x\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.355885 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.356132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.356653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.356663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.356697 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.356978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.356776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.358451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.358419 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.358783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.358766 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.414821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.414787 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.414986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.414956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq48x\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.492136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.492051 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:42.932320 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.932289 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74"] Nov 02 15:20:42.936263 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:20:42.936236 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b665f2_bce6_4a65_a23e_d092f0e271bc.slice/crio-14351f2bbf5115ab2308e05637892436338e9622bfdbb88a07ae86118566615f WatchSource:0}: Error finding container 14351f2bbf5115ab2308e05637892436338e9622bfdbb88a07ae86118566615f: Status 404 returned error can't find the container with id 14351f2bbf5115ab2308e05637892436338e9622bfdbb88a07ae86118566615f Nov 02 15:20:42.939147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.939104 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:42.939263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.939195 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:42.939263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:42.939245 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:43.949540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:43.948766 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" event={"ID":"d3b665f2-bce6-4a65-a23e-d092f0e271bc","Type":"ContainerStarted","Data":"c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11"} Nov 02 15:20:43.949540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:43.948833 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" event={"ID":"d3b665f2-bce6-4a65-a23e-d092f0e271bc","Type":"ContainerStarted","Data":"14351f2bbf5115ab2308e05637892436338e9622bfdbb88a07ae86118566615f"} Nov 02 15:20:44.493194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:44.493152 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:44.495196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:44.495167 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:20:44.495350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:44.495225 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 15:20:45.493410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:45.493374 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:20:45.493870 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:45.493442 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 15:20:46.492751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:46.492717 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:20:46.492933 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:46.492782 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 15:20:47.493447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:47.493408 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:20:47.493997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:47.493473 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 15:20:48.493281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:48.493244 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:20:48.493442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:48.493296 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 15:20:49.493157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:49.493125 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 02 15:20:49.493545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:49.493191 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 02 15:20:50.497561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:50.497498 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:50.498019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:50.497883 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:50.498560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:50.498503 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:20:50.718166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:50.718081 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podStartSLOduration=8.718067733 podStartE2EDuration="8.718067733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:44.170775174 +0000 UTC m=+1398.335327714" watchObservedRunningTime="2026-11-02 15:20:50.718067733 +0000 UTC m=+1404.882620272" Nov 02 15:20:57.339579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.339542 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb"] Nov 02 15:20:57.345627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.345605 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.434993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.434946 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb"] Nov 02 15:20:57.470153 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470295 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdvc\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470395 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470440 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470505 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470549 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470581 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.470770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.470683 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.571858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.571821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.571888 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.571957 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdvc\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572027 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572066 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572124 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572150 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572224 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572281 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572580 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572627 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572646 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.572889 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.572859 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.574412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.574390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.574760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.574738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.633929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.633895 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.634194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.634169 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdvc\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc\") pod \"e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.662878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.662853 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:57.935686 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:20:57.935605 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:20:57.967465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.967428 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c"] Nov 02 15:20:57.975103 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:57.975070 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.027177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.027140 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:20:58.027389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.027263 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:20:58.027490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.027472 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c"] Nov 02 15:20:58.081289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.081252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.081485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.081405 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zld4v\" (UniqueName: \"kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.081485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.081462 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.102114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.102085 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb"] Nov 02 15:20:58.102276 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:20:58.102258 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc090c586_ac91_41db_834c_629b79432c1f.slice/crio-fe004718aa0ffbfedacafdf47eff418f4293e106d5885b17c12f3dccdf20c6ed WatchSource:0}: Error finding container fe004718aa0ffbfedacafdf47eff418f4293e106d5885b17c12f3dccdf20c6ed: Status 404 returned error can't find the container with id fe004718aa0ffbfedacafdf47eff418f4293e106d5885b17c12f3dccdf20c6ed Nov 02 15:20:58.104740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.104704 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.104830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.104763 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.104830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.104790 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:20:58.183255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.183229 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zld4v\" (UniqueName: \"kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.183364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.183276 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.183477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.183459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.184172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.184138 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.186065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.186013 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.243400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.243365 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zld4v\" (UniqueName: \"kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v\") pod \"maas-api-revoke-sub-7842d4ad-l555c\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.290468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.290433 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:20:58.675557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:58.675140 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c"] Nov 02 15:20:58.678462 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:20:58.678425 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod770968a3_5b11_4dc1_b1ef_9ad745d32c9a.slice/crio-c9832ca74d9e8d9ddc0610cc199744f73bcdd1f18c1ef63cbf57cacd04bed074 WatchSource:0}: Error finding container c9832ca74d9e8d9ddc0610cc199744f73bcdd1f18c1ef63cbf57cacd04bed074: Status 404 returned error can't find the container with id c9832ca74d9e8d9ddc0610cc199744f73bcdd1f18c1ef63cbf57cacd04bed074 Nov 02 15:20:59.068959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.068928 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" event={"ID":"c090c586-ac91-41db-834c-629b79432c1f","Type":"ContainerStarted","Data":"d3e47317ffe9d95877fbe1c2498bf0a651841f9ba5585b3c3fcb9efba8efd3c3"} Nov 02 15:20:59.069088 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.068992 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" event={"ID":"c090c586-ac91-41db-834c-629b79432c1f","Type":"ContainerStarted","Data":"fe004718aa0ffbfedacafdf47eff418f4293e106d5885b17c12f3dccdf20c6ed"} Nov 02 15:20:59.070681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.070647 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" event={"ID":"770968a3-5b11-4dc1-b1ef-9ad745d32c9a","Type":"ContainerStarted","Data":"c9832ca74d9e8d9ddc0610cc199744f73bcdd1f18c1ef63cbf57cacd04bed074"} Nov 02 15:20:59.247118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.247027 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podStartSLOduration=2.24700788 podStartE2EDuration="2.24700788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:20:59.245835644 +0000 UTC m=+1413.410388185" watchObservedRunningTime="2026-11-02 15:20:59.24700788 +0000 UTC m=+1413.411560422" Nov 02 15:20:59.666227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.666185 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:20:59.666443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.666315 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 15:20:59.666443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:20:59.666359 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 15:21:00.081316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:00.081221 5091 generic.go:356] "Generic (PLEG): container finished" podID="770968a3-5b11-4dc1-b1ef-9ad745d32c9a" containerID="1f169c68d4c5a9f51bc74411f7f7e703acd4c213c922bf39f29faff04e3ab8c4" exitCode=0 Nov 02 15:21:00.081803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:00.081307 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" event={"ID":"770968a3-5b11-4dc1-b1ef-9ad745d32c9a","Type":"ContainerDied","Data":"1f169c68d4c5a9f51bc74411f7f7e703acd4c213c922bf39f29faff04e3ab8c4"} Nov 02 15:21:00.663968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:00.663932 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 15:21:00.664174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:00.663999 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 15:21:01.304325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.304305 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:21:01.324205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.324181 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca\") pod \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " Nov 02 15:21:01.324344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.324331 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v\" (UniqueName: \"kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v\") pod \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " Nov 02 15:21:01.324396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.324371 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca\") pod \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\" (UID: \"770968a3-5b11-4dc1-b1ef-9ad745d32c9a\") " Nov 02 15:21:01.324751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.324727 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca" pod "770968a3-5b11-4dc1-b1ef-9ad745d32c9a" (UID: "770968a3-5b11-4dc1-b1ef-9ad745d32c9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:01.326188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.326167 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca" pod "770968a3-5b11-4dc1-b1ef-9ad745d32c9a" (UID: "770968a3-5b11-4dc1-b1ef-9ad745d32c9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:01.326316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.326300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v" pod "770968a3-5b11-4dc1-b1ef-9ad745d32c9a" (UID: "770968a3-5b11-4dc1-b1ef-9ad745d32c9a"). InnerVolumeSpecName "kube-api-access-zld4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:01.425708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.425677 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zld4v\" (UniqueName: \"kubernetes.io/projected/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-kube-api-access-zld4v\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:01.425708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.425705 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:01.425708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.425715 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/770968a3-5b11-4dc1-b1ef-9ad745d32c9a-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:01.663541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.663430 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 15:21:01.663541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:01.663478 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 15:21:02.095801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:02.095767 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" Nov 02 15:21:02.095801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:02.095792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c" event={"ID":"770968a3-5b11-4dc1-b1ef-9ad745d32c9a","Type":"ContainerDied","Data":"c9832ca74d9e8d9ddc0610cc199744f73bcdd1f18c1ef63cbf57cacd04bed074"} Nov 02 15:21:02.096014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:02.095839 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9832ca74d9e8d9ddc0610cc199744f73bcdd1f18c1ef63cbf57cacd04bed074" Nov 02 15:21:02.663850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:02.663812 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 15:21:02.664227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:02.663883 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 15:21:03.663502 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:03.663457 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 15:21:03.663798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:03.663554 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 15:21:04.664001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:04.663963 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 02 15:21:04.664403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:04.664032 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 02 15:21:05.668450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:05.668415 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:21:05.669154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:05.669133 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:21:05.669906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:05.669884 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:21:09.632542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:09.624615 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:09.701651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:09.701601 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:09.798659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:09.798609 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:09.891917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:09.891817 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.079550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.075615 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.188597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.185618 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.207732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.207686 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.208309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.208281 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded" Nov 02 15:21:10.208501 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.208481 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:10.275379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.275338 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 15:21:10.276039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.275657 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.634540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.627609 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.634540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.628359 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:21:10.707711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.707656 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 15:21:10.709442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.709418 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:21:10.892386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.892248 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:10.893644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.893621 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:21:10.991578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:10.969689 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.195738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.195628 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.228550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.205438 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:21:11.265665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.265619 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.294556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.267175 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 15:21:11.355540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.295883 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:21:11.355540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.328840 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:21:11.355540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.328423 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.355540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.338672 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 15:21:11.355540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.339403 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:11.355540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.339462 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:21:11.416542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.405011 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:21:11.497589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.494125 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:21:11.644186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.644156 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:11.666642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.666607 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:21:11.706590 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.680851 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:11.714395 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.710346 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:21:11.722395 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:11.722370 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:21:14.350477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.350440 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp"] Nov 02 15:21:14.351235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.351212 5091 cpu_manager.go:414] "containerMap: removing container" podUID="770968a3-5b11-4dc1-b1ef-9ad745d32c9a" containerName="revoke-keys" Nov 02 15:21:14.351336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.351235 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="770968a3-5b11-4dc1-b1ef-9ad745d32c9a" containerName="revoke-keys" Nov 02 15:21:14.351443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.351430 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="770968a3-5b11-4dc1-b1ef-9ad745d32c9a" containerName="revoke-keys" Nov 02 15:21:14.355066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.355045 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.385942 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.385912 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7gwc\" (UniqueName: \"kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.386100 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.386068 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.386152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.386120 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.419277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.419237 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp"] Nov 02 15:21:14.452362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.452326 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:21:14.452656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.452506 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:21:14.487241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.487202 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.487429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.487268 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.487429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.487408 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7gwc\" (UniqueName: \"kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.488086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.488064 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.489833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.489805 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.546202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.546165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7gwc\" (UniqueName: \"kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc\") pod \"maas-api-revoke-sub-65c32e32-xf6cp\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:14.672867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:14.672828 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:15.056953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:15.056867 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp"] Nov 02 15:21:15.060414 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:15.060378 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb183bed4_6983_409a_8f10_82d7ec35b33f.slice/crio-cd6e928ca28db429f6ff3a08baf862e7d64b4f64d1cf67bd533fd3fa45846757 WatchSource:0}: Error finding container cd6e928ca28db429f6ff3a08baf862e7d64b4f64d1cf67bd533fd3fa45846757: Status 404 returned error can't find the container with id cd6e928ca28db429f6ff3a08baf862e7d64b4f64d1cf67bd533fd3fa45846757 Nov 02 15:21:15.504143 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:15.504105 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" event={"ID":"b183bed4-6983-409a-8f10-82d7ec35b33f","Type":"ContainerStarted","Data":"18dbacdff221cf631b39676456c4c2a090d7a4196b8445c69922f0e6dbcd0c9d"} Nov 02 15:21:15.504606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:15.504159 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" event={"ID":"b183bed4-6983-409a-8f10-82d7ec35b33f","Type":"ContainerStarted","Data":"cd6e928ca28db429f6ff3a08baf862e7d64b4f64d1cf67bd533fd3fa45846757"} Nov 02 15:21:15.620956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:15.620907 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" podStartSLOduration=1.62089431 podStartE2EDuration="1.62089431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:15.62033559 +0000 UTC m=+1429.784888129" watchObservedRunningTime="2026-11-02 15:21:15.62089431 +0000 UTC m=+1429.785446851" Nov 02 15:21:16.513404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:16.513365 5091 generic.go:356] "Generic (PLEG): container finished" podID="b183bed4-6983-409a-8f10-82d7ec35b33f" containerID="18dbacdff221cf631b39676456c4c2a090d7a4196b8445c69922f0e6dbcd0c9d" exitCode=0 Nov 02 15:21:16.513840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:16.513451 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" event={"ID":"b183bed4-6983-409a-8f10-82d7ec35b33f","Type":"ContainerDied","Data":"18dbacdff221cf631b39676456c4c2a090d7a4196b8445c69922f0e6dbcd0c9d"} Nov 02 15:21:17.282097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.282051 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf"] Nov 02 15:21:17.288632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.287213 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.343350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.343314 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf"] Nov 02 15:21:17.425210 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.425172 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmgc\" (UniqueName: \"kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.425381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.425241 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.425381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.425286 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.527438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.527402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmgc\" (UniqueName: \"kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.527950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.527930 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.528082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.528063 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.528732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.528706 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.530907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.530884 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.587067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.587004 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmgc\" (UniqueName: \"kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc\") pod \"maas-api-revoke-sub-db90c09f-dspcf\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.604730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.604700 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:17.690005 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.689602 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:17.730683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.730648 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca\") pod \"b183bed4-6983-409a-8f10-82d7ec35b33f\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " Nov 02 15:21:17.730863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.730745 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc\" (UniqueName: \"kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc\") pod \"b183bed4-6983-409a-8f10-82d7ec35b33f\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " Nov 02 15:21:17.730863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.730774 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca\") pod \"b183bed4-6983-409a-8f10-82d7ec35b33f\" (UID: \"b183bed4-6983-409a-8f10-82d7ec35b33f\") " Nov 02 15:21:17.731573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.731549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca" pod "b183bed4-6983-409a-8f10-82d7ec35b33f" (UID: "b183bed4-6983-409a-8f10-82d7ec35b33f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:17.733863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.733462 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc" pod "b183bed4-6983-409a-8f10-82d7ec35b33f" (UID: "b183bed4-6983-409a-8f10-82d7ec35b33f"). InnerVolumeSpecName "kube-api-access-w7gwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:17.733863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.733564 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca" pod "b183bed4-6983-409a-8f10-82d7ec35b33f" (UID: "b183bed4-6983-409a-8f10-82d7ec35b33f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:17.841630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.841540 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.841630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.841578 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7gwc\" (UniqueName: \"kubernetes.io/projected/b183bed4-6983-409a-8f10-82d7ec35b33f-kube-api-access-w7gwc\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:17.841630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:17.841595 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b183bed4-6983-409a-8f10-82d7ec35b33f-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:18.013837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.013807 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf"] Nov 02 15:21:18.017132 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:18.017103 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b678f54_4c83_4723_887d_ece04991a117.slice/crio-ee9a18683602628065bec32ea869f1786374253102f441420b35041a935da802 WatchSource:0}: Error finding container ee9a18683602628065bec32ea869f1786374253102f441420b35041a935da802: Status 404 returned error can't find the container with id ee9a18683602628065bec32ea869f1786374253102f441420b35041a935da802 Nov 02 15:21:18.529669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.529557 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" event={"ID":"b183bed4-6983-409a-8f10-82d7ec35b33f","Type":"ContainerDied","Data":"cd6e928ca28db429f6ff3a08baf862e7d64b4f64d1cf67bd533fd3fa45846757"} Nov 02 15:21:18.529669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.529633 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6e928ca28db429f6ff3a08baf862e7d64b4f64d1cf67bd533fd3fa45846757" Nov 02 15:21:18.529669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.529573 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp" Nov 02 15:21:18.532971 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.532941 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" event={"ID":"5b678f54-4c83-4723-887d-ece04991a117","Type":"ContainerStarted","Data":"859cdee56dea98af8420afca866c77e2ac19cb94f5447532db21e33d60e321bd"} Nov 02 15:21:18.533095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.532998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" event={"ID":"5b678f54-4c83-4723-887d-ece04991a117","Type":"ContainerStarted","Data":"ee9a18683602628065bec32ea869f1786374253102f441420b35041a935da802"} Nov 02 15:21:18.653125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:18.653069 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" podStartSLOduration=1.653056868 podStartE2EDuration="1.653056868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:18.652499989 +0000 UTC m=+1432.817052529" watchObservedRunningTime="2026-11-02 15:21:18.653056868 +0000 UTC m=+1432.817609411" Nov 02 15:21:19.542708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:19.542672 5091 generic.go:356] "Generic (PLEG): container finished" podID="5b678f54-4c83-4723-887d-ece04991a117" containerID="859cdee56dea98af8420afca866c77e2ac19cb94f5447532db21e33d60e321bd" exitCode=0 Nov 02 15:21:19.543208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:19.542767 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" event={"ID":"5b678f54-4c83-4723-887d-ece04991a117","Type":"ContainerDied","Data":"859cdee56dea98af8420afca866c77e2ac19cb94f5447532db21e33d60e321bd"} Nov 02 15:21:20.718512 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.718487 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:20.782842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.782810 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc\" (UniqueName: \"kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc\") pod \"5b678f54-4c83-4723-887d-ece04991a117\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " Nov 02 15:21:20.783010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.782944 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca\") pod \"5b678f54-4c83-4723-887d-ece04991a117\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " Nov 02 15:21:20.783123 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.783099 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca\") pod \"5b678f54-4c83-4723-887d-ece04991a117\" (UID: \"5b678f54-4c83-4723-887d-ece04991a117\") " Nov 02 15:21:20.783493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.783467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca" pod "5b678f54-4c83-4723-887d-ece04991a117" (UID: "5b678f54-4c83-4723-887d-ece04991a117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:20.783809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.783787 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b678f54-4c83-4723-887d-ece04991a117-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:20.785217 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.785187 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca" pod "5b678f54-4c83-4723-887d-ece04991a117" (UID: "5b678f54-4c83-4723-887d-ece04991a117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:20.785417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.785400 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc" pod "5b678f54-4c83-4723-887d-ece04991a117" (UID: "5b678f54-4c83-4723-887d-ece04991a117"). InnerVolumeSpecName "kube-api-access-qvmgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:20.885494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.885457 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvmgc\" (UniqueName: \"kubernetes.io/projected/5b678f54-4c83-4723-887d-ece04991a117-kube-api-access-qvmgc\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:20.885494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:20.885490 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b678f54-4c83-4723-887d-ece04991a117-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:21.557543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:21.557493 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" event={"ID":"5b678f54-4c83-4723-887d-ece04991a117","Type":"ContainerDied","Data":"ee9a18683602628065bec32ea869f1786374253102f441420b35041a935da802"} Nov 02 15:21:21.557745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:21.557570 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9a18683602628065bec32ea869f1786374253102f441420b35041a935da802" Nov 02 15:21:21.557745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:21.557506 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf" Nov 02 15:21:26.056076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:26.056038 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt"] Nov 02 15:21:26.102924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:26.102892 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b88fef46-bbswt"] Nov 02 15:21:27.302093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302055 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq"] Nov 02 15:21:27.302640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302625 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b183bed4-6983-409a-8f10-82d7ec35b33f" containerName="revoke-keys" Nov 02 15:21:27.302691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302641 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b183bed4-6983-409a-8f10-82d7ec35b33f" containerName="revoke-keys" Nov 02 15:21:27.302691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302682 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5b678f54-4c83-4723-887d-ece04991a117" containerName="revoke-keys" Nov 02 15:21:27.302691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302687 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b678f54-4c83-4723-887d-ece04991a117" containerName="revoke-keys" Nov 02 15:21:27.302804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302794 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b183bed4-6983-409a-8f10-82d7ec35b33f" containerName="revoke-keys" Nov 02 15:21:27.302841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.302808 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b678f54-4c83-4723-887d-ece04991a117" containerName="revoke-keys" Nov 02 15:21:27.307668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.307642 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378112 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378386 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kss\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378423 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378558 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378636 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378696 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378719 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.378958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.379087 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.378990 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480411 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kss\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480508 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480552 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480594 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480685 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.480720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480712 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480876 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.480944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.481113 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481332 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.481308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.481423 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.481615 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.481703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.481679 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.482961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.482935 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.483187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.483164 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.518558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.518508 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq"] Nov 02 15:21:27.540257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.540225 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kss\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.540665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.540645 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token\") pod \"e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.627419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.627384 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:27.935358 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:21:27.935273 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:21:27.961660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:27.961627 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="708fd46c-763b-43fd-a6da-e50eaf3a8271" path="/var/lib/kubelet/pods/708fd46c-763b-43fd-a6da-e50eaf3a8271/volumes" Nov 02 15:21:28.067883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.067848 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq"] Nov 02 15:21:28.105636 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:28.105600 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b25ebf_afb3_49d6_b7cb_e9eca9880ccd.slice/crio-0f5bbd88bcd8eb37856a018cca656a8de80e00e766624db5abcfa45401186176 WatchSource:0}: Error finding container 0f5bbd88bcd8eb37856a018cca656a8de80e00e766624db5abcfa45401186176: Status 404 returned error can't find the container with id 0f5bbd88bcd8eb37856a018cca656a8de80e00e766624db5abcfa45401186176 Nov 02 15:21:28.108276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.108242 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.108349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.108311 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.108349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.108343 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:28.624195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.624120 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" event={"ID":"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd","Type":"ContainerStarted","Data":"188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79"} Nov 02 15:21:28.624195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.624184 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" event={"ID":"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd","Type":"ContainerStarted","Data":"0f5bbd88bcd8eb37856a018cca656a8de80e00e766624db5abcfa45401186176"} Nov 02 15:21:28.628608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.628583 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:28.792880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:28.792804 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podStartSLOduration=1.792789062 podStartE2EDuration="1.792789062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:28.789615327 +0000 UTC m=+1442.954167870" watchObservedRunningTime="2026-11-02 15:21:28.792789062 +0000 UTC m=+1442.957341624" Nov 02 15:21:29.158968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.158932 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5"] Nov 02 15:21:29.163328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.163304 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202221 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202425 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202279 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202425 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202399 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202607 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202636 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202706 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.202873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.202734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc8c8\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.221047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.221017 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5"] Nov 02 15:21:29.303732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.303685 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.303927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.303775 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.303927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.303897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.303940 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.303968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.303997 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.304069 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.304100 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc8c8\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.304169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.304997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.304970 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.305257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.305236 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.305492 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.305473 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.305736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.305716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.305898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.305872 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.307750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.307731 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.309658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.309636 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.381486 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.381449 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.381660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.381641 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc8c8\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-679p5\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.478783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.478703 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:29.568477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.568026 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb"] Nov 02 15:21:29.576555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.576532 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610080 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610200 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610380 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610503 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610597 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610621 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610758 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610739 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzhz\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.610827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.610811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.611024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.611001 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.617429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.617404 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf"] Nov 02 15:21:29.621734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.621717 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.657759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.640735 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:29.657759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.640787 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:29.675199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.675171 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb"] Nov 02 15:21:29.690760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.686541 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf"] Nov 02 15:21:29.714288 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714338 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kl4w\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714409 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714436 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714470 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714496 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714597 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714635 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714671 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714747 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714772 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714819 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.714949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714870 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzhz\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.714949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714904 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.715251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.714985 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.715251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.715019 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.715251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.715123 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.715969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.715942 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.716117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.716009 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.716966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.716941 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.717125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.717099 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.717795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.717768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.719872 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.719846 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.720051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.720034 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.784751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.784685 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzhz\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.785058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.785039 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.816553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816498 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.816680 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.816680 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816617 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.816772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816741 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kl4w\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.816824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816809 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.816869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816840 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.816941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816944 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816919 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.817000 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.817088 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.816999 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817433 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.817411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.817676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.817659 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.819115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.819086 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.819383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.819361 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.877454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.877414 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.881248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.881221 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kl4w\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w\") pod \"e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:29.900190 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.900159 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:29.968305 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:29.968271 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f714a30_f65a_4712_a5ad_90015f83a450.slice/crio-0ad526d61b54711e3656718cdbcd7e1403a62fa15d58873a536ce4509ff47122 WatchSource:0}: Error finding container 0ad526d61b54711e3656718cdbcd7e1403a62fa15d58873a536ce4509ff47122: Status 404 returned error can't find the container with id 0ad526d61b54711e3656718cdbcd7e1403a62fa15d58873a536ce4509ff47122 Nov 02 15:21:29.972576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.972533 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.972684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.972635 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.972684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.972677 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:29.977136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.977116 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5"] Nov 02 15:21:29.977573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:29.977538 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:30.323355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.321250 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb"] Nov 02 15:21:30.363646 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.363611 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.363898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.363877 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.364018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.364006 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.439831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.426203 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf"] Nov 02 15:21:30.464391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.464355 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.464614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.464591 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.465013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.464993 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:30.630541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.629940 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:30.630541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.629998 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:30.699738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.699636 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" event={"ID":"1f714a30-f65a-4712-a5ad-90015f83a450","Type":"ContainerStarted","Data":"375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59"} Nov 02 15:21:30.699738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.699700 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" event={"ID":"1f714a30-f65a-4712-a5ad-90015f83a450","Type":"ContainerStarted","Data":"0ad526d61b54711e3656718cdbcd7e1403a62fa15d58873a536ce4509ff47122"} Nov 02 15:21:30.704562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.702700 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" event={"ID":"24b580d5-d425-4fca-8453-7ea6e61e2a75","Type":"ContainerStarted","Data":"c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3"} Nov 02 15:21:30.704562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.702752 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" event={"ID":"24b580d5-d425-4fca-8453-7ea6e61e2a75","Type":"ContainerStarted","Data":"2e268c8b9c9017b35f14c9e22834bb8ed3456351b0cf3e51ad314d002083a95d"} Nov 02 15:21:30.708547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.706952 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" event={"ID":"c57a0c79-3d61-40ef-a0bd-5d8eea34454f","Type":"ContainerStarted","Data":"38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5"} Nov 02 15:21:30.708547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.707008 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" event={"ID":"c57a0c79-3d61-40ef-a0bd-5d8eea34454f","Type":"ContainerStarted","Data":"7d3d7454901563abbc8d2512787d547aec669ebd68e120dd7c5f03b4e3ab5db0"} Nov 02 15:21:30.825833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.825760 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podStartSLOduration=1.825738149 podStartE2EDuration="1.825738149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:30.821423757 +0000 UTC m=+1444.985976297" watchObservedRunningTime="2026-11-02 15:21:30.825738149 +0000 UTC m=+1444.990290691" Nov 02 15:21:30.907541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.904727 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:30.964539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.964434 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podStartSLOduration=1.964413754 podStartE2EDuration="1.964413754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:30.942215872 +0000 UTC m=+1445.106768413" watchObservedRunningTime="2026-11-02 15:21:30.964413754 +0000 UTC m=+1445.128966295" Nov 02 15:21:30.985540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:30.978691 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:31.102540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.102128 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podStartSLOduration=2.102104573 podStartE2EDuration="2.102104573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:31.078867347 +0000 UTC m=+1445.243419897" watchObservedRunningTime="2026-11-02 15:21:31.102104573 +0000 UTC m=+1445.266657114" Nov 02 15:21:31.486832 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.486799 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:31.487189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.487165 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.487375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.487351 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:31.640539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.634627 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.640539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.634685 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:31.902326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.901981 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:31.902326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:31.902063 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:32.022736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.022609 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.022736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.022668 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:32.123540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.108137 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6"] Nov 02 15:21:32.159017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.158921 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47a351ad-2w2n6"] Nov 02 15:21:32.500601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.487024 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.500601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.487858 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:32.631733 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.631694 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:32.632023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.631994 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:32.669766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.669713 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:32.779442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:32.766595 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.105706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.052817 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.105706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.052873 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:33.105706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.053700 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.105706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.062722 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.105706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.062868 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:33.247792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.247746 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.345441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.327434 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.503662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.503612 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.544539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.537185 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.561945 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.560829 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.561945 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.560892 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:33.588542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.563291 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.665743 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.665697 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.692934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.666279 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.693641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.693611 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:33.693830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.692571 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.695568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.695546 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:21:33.723644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.722605 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:33.961206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.961169 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:33.961529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:33.961487 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:34.010801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.007651 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.010801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.007721 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:34.069981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.069634 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:34.071077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.070614 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.099537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.098736 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.321591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.309992 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.328223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.328176 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.328618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.328381 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:21:34.336468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.329622 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.417277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.417234 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.473541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.440446 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08d279e-fe25-49da-ab6c-8d83c0894b8e" path="/var/lib/kubelet/pods/c08d279e-fe25-49da-ab6c-8d83c0894b8e/volumes" Nov 02 15:21:34.474389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.474359 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:21:34.475984 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.475959 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:21:34.476177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.476163 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:21:34.511561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.511501 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.523656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.523625 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:21:34.560540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.534988 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.560540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.535053 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:34.560540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.552330 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.560540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.552756 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:21:34.636544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.635076 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:34.636544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.635162 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:34.715501 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.715441 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.771574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.721647 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:34.790390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.790348 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:34.794654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.732635 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:34.828149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:34.828118 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:21:35.031598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.030342 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.031598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.030396 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:35.031598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.030688 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.031598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.030725 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:35.073654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.073605 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:21:35.086540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.075180 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.136898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.085649 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:21:35.146727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.103592 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.207542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.173474 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:21:35.311795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.311670 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded" Nov 02 15:21:35.361566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.338386 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:21:35.361566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.338780 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.361566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.339026 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.361566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.339286 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:21:35.493253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.493126 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.497662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.493509 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:35.532297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.532254 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:21:35.781111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.587242 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.782386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.635199 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.782594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.782569 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:35.782709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.780692 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:35.783761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.756328 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.808862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.808821 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:35.955371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.955333 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:35.962682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.962648 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:35.989311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:35.989278 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.011540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.004641 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:36.088766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.084458 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:21:36.152213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.151957 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:21:36.221550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.181291 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.365705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.342866 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.370587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.349723 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:36.371301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.349778 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:21:36.374062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.371589 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.640588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.637471 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.640588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.637546 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 02 15:21:36.640588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.638361 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:21:36.640588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.638430 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:21:36.669564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.658365 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.669564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.658426 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:36.782831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.782787 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.802678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.787612 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.816710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.816664 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.817447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.817111 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:36.930085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.929987 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.930262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.930095 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:36.989544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.983952 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:36.989544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:36.984015 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:37.160179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.160044 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.199631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.189144 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.199993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.199938 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.394542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.369885 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:37.395654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.372620 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.425186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.373801 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.545702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.487810 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.560185 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.560140 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:37.642663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.642615 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.721164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.719414 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:21:37.721164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.720024 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:21:37.721855 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.721835 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:21:37.732064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.732027 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.803603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.797269 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:37.803603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.798697 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:37.808564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.804338 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.837540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.814614 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:21:37.837540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.817743 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.837540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.825740 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:37.902896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.902867 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:21:37.904262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.904218 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:37.904451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:37.904428 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:38.141955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.114531 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.141955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.114581 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:38.190007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.189657 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.201735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.201694 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.313275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.313231 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:21:38.336923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.336895 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:38.397116 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.397018 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.569246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.556933 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:21:38.569246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.558480 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:38.634254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.634221 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:21:38.642596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.642567 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.642830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.642807 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 02 15:21:38.668782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.668069 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:21:38.682020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.681980 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:21:38.693612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.693014 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:21:38.915648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.915612 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.915937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.915905 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 02 15:21:38.981331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.981242 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Nov 02 15:21:38.984493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:38.984454 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Nov 02 15:21:39.489668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:39.489631 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:39.491236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:39.491210 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:39.492357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:39.492332 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:21:40.342893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:40.342743 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:40.343806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:40.343580 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:40.345672 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:40.345653 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:21:40.585106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:40.585071 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:40.586121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:40.586094 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:40.587229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:40.587208 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:21:41.410154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.410125 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22"] Nov 02 15:21:41.415434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.415413 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.465563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.465477 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:21:41.465704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.465607 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:21:41.469920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.469487 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22"] Nov 02 15:21:41.488115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.488016 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.488315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.488267 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.488469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.488414 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7b4r\" (UniqueName: \"kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.590279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.590244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7b4r\" (UniqueName: \"kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.590777 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.590320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.590777 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.590412 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.591158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.591133 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.592846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.592827 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.649195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.649165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7b4r\" (UniqueName: \"kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r\") pod \"maas-api-revoke-sub-81ef4479-v9g22\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:41.730474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:41.730386 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:42.050369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.050281 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8"] Nov 02 15:21:42.054345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.054316 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c67ea374-rn4p8"] Nov 02 15:21:42.096944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.096915 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m"] Nov 02 15:21:42.100203 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.100181 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-271a0cc5-26d8m"] Nov 02 15:21:42.164168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.164135 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22"] Nov 02 15:21:42.168409 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:42.168381 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3760e0_2421_4c88_a408_2200536074e1.slice/crio-10b83fe05eeede06b36d0a6cdd2236e9b9786fa880a59f04390665ded02364aa WatchSource:0}: Error finding container 10b83fe05eeede06b36d0a6cdd2236e9b9786fa880a59f04390665ded02364aa: Status 404 returned error can't find the container with id 10b83fe05eeede06b36d0a6cdd2236e9b9786fa880a59f04390665ded02364aa Nov 02 15:21:42.931854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.931820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" event={"ID":"7b3760e0-2421-4c88-a408-2200536074e1","Type":"ContainerStarted","Data":"12ae423b874dbc1e036a8ce79888e7aabbb5b749db4a88d07cc1de63e91431a8"} Nov 02 15:21:42.932373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:42.931910 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" event={"ID":"7b3760e0-2421-4c88-a408-2200536074e1","Type":"ContainerStarted","Data":"10b83fe05eeede06b36d0a6cdd2236e9b9786fa880a59f04390665ded02364aa"} Nov 02 15:21:43.042418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:43.042366 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" podStartSLOduration=2.042353444 podStartE2EDuration="2.042353444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:43.042082006 +0000 UTC m=+1457.206634549" watchObservedRunningTime="2026-11-02 15:21:43.042353444 +0000 UTC m=+1457.206905984" Nov 02 15:21:43.941721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:43.941681 5091 generic.go:356] "Generic (PLEG): container finished" podID="7b3760e0-2421-4c88-a408-2200536074e1" containerID="12ae423b874dbc1e036a8ce79888e7aabbb5b749db4a88d07cc1de63e91431a8" exitCode=0 Nov 02 15:21:43.942144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:43.941756 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" event={"ID":"7b3760e0-2421-4c88-a408-2200536074e1","Type":"ContainerDied","Data":"12ae423b874dbc1e036a8ce79888e7aabbb5b749db4a88d07cc1de63e91431a8"} Nov 02 15:21:43.960892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:43.960865 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0132b239-fec0-426a-b436-db370fd947fd" path="/var/lib/kubelet/pods/0132b239-fec0-426a-b436-db370fd947fd/volumes" Nov 02 15:21:43.961336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:43.961322 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930382b7-1e4a-44d7-b4b0-51cab6e94e10" path="/var/lib/kubelet/pods/930382b7-1e4a-44d7-b4b0-51cab6e94e10/volumes" Nov 02 15:21:44.355158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.353242 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb"] Nov 02 15:21:44.362494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.362466 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.469919 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.469881 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb"] Nov 02 15:21:44.536336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536299 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536358 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwr8p\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536390 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536531 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536683 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536731 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536760 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.536799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.536792 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.652922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.652873 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.655671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.655640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwr8p\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.656197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.656173 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.656730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.656711 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.657237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.657219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.657890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.657869 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.658376 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.658357 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.658834 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.658806 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.659402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.659383 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.688121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.658266 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.688363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.657109 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.688467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.656638 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.688577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.659810 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.688677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.655097 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.694064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.694017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.694822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.694789 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.717392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.717322 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwr8p\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:44.722641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:44.718969 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token\") pod \"e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:45.012593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:45.002860 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:46.400605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.399645 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.575660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.575613 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:21:46.674596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.671649 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:21:46.731831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.731789 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.759691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.759644 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:46.836012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.819740 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:21:46.836012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.820099 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 15:21:46.836012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.820355 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:21:46.866896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:46.866833 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.322915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.322873 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.323411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.323377 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.419600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.417191 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:47.419600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.417991 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:21:47.581106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.581008 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:21:47.582158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.582131 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:21:47.676542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.647602 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.684561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.676812 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.736027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.732179 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.737477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.737449 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:21:47.737986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.737968 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:21:47.779701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.779652 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.781308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.781282 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:21:47.847240 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.844630 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.847240 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.846085 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:21:47.848811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.847651 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.849117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.848988 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:47.849117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.849070 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:21:47.849425 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.849289 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:21:47.890221 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.874584 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:47.894537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:47.894247 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:21:48.060630 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:48.022622 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c74967_4c36_4996_ba34_099e94f3f53a.slice/crio-dda312f6b0fa0487475c04c922f4a91c389fcb6609e467a6e56fa29e4fa264f2 WatchSource:0}: Error finding container dda312f6b0fa0487475c04c922f4a91c389fcb6609e467a6e56fa29e4fa264f2: Status 404 returned error can't find the container with id dda312f6b0fa0487475c04c922f4a91c389fcb6609e467a6e56fa29e4fa264f2 Nov 02 15:21:48.150594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.148796 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:48.150594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.148891 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:48.150594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.148936 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:21:48.181834 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.171032 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb"] Nov 02 15:21:48.305537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.300810 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:48.337232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.334185 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.337232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.334658 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:21:48.457657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.455022 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:48.608863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.590611 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:21:48.832873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.831818 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:21:48.832873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.832350 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:48.910030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.883647 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 15:21:48.910030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.909605 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:48.922571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.914595 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:21:48.922571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.918622 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 02 15:21:48.922571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:48.920359 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.070550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.064636 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:49.070550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.064723 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.219543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.212459 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" event={"ID":"7b3760e0-2421-4c88-a408-2200536074e1","Type":"ContainerDied","Data":"10b83fe05eeede06b36d0a6cdd2236e9b9786fa880a59f04390665ded02364aa"} Nov 02 15:21:49.219543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.212544 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10b83fe05eeede06b36d0a6cdd2236e9b9786fa880a59f04390665ded02364aa" Nov 02 15:21:49.252543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.240132 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:49.314543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.301769 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.344568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.326073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" event={"ID":"68c74967-4c36-4996-ba34-099e94f3f53a","Type":"ContainerStarted","Data":"dda312f6b0fa0487475c04c922f4a91c389fcb6609e467a6e56fa29e4fa264f2"} Nov 02 15:21:49.353011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.335559 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.367590 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.335591 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:21:49.368413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.364945 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca\") pod \"7b3760e0-2421-4c88-a408-2200536074e1\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " Nov 02 15:21:49.391584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.368829 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca\") pod \"7b3760e0-2421-4c88-a408-2200536074e1\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " Nov 02 15:21:49.391903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.391881 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r\" (UniqueName: \"kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r\") pod \"7b3760e0-2421-4c88-a408-2200536074e1\" (UID: \"7b3760e0-2421-4c88-a408-2200536074e1\") " Nov 02 15:21:49.392894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.382168 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca" pod "7b3760e0-2421-4c88-a408-2200536074e1" (UID: "7b3760e0-2421-4c88-a408-2200536074e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:49.436836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.434995 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:49.469845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.459782 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.605554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.579811 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca" pod "7b3760e0-2421-4c88-a408-2200536074e1" (UID: "7b3760e0-2421-4c88-a408-2200536074e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:49.605554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.597880 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:49.649628 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.646630 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b3760e0-2421-4c88-a408-2200536074e1-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:49.649947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.649909 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r" pod "7b3760e0-2421-4c88-a408-2200536074e1" (UID: "7b3760e0-2421-4c88-a408-2200536074e1"). InnerVolumeSpecName "kube-api-access-c7b4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:49.927889 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.927762 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 15:21:49.941820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.941780 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:21:50.009615 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.930487 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7b4r\" (UniqueName: \"kubernetes.io/projected/7b3760e0-2421-4c88-a408-2200536074e1-kube-api-access-c7b4r\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:50.013573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.939509 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.017536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.928162 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 15:21:50.037587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:49.966327 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Nov 02 15:21:50.079934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.039401 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.222593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.052729 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.244370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.179671 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:21:50.244721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.244686 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.245017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.244982 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.353456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.352419 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.373999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.372941 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.408495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.374629 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.466547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.466184 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:21:50.639598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.612430 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:50.685586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.678178 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:50.700656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.700630 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:21:50.719142 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.719120 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:21:50.719329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.719319 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:21:50.719442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.719426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" event={"ID":"68c74967-4c36-4996-ba34-099e94f3f53a","Type":"ContainerStarted","Data":"1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949"} Nov 02 15:21:50.719565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.706242 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22" Nov 02 15:21:50.813020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:50.812987 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:21:51.025626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.025540 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:21:51.069536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.067102 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:51.095697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.095651 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.096668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.096638 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:21:51.120383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.118695 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:21:51.134152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.134051 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:21:51.135932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.135493 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:21:51.168657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.158952 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:21:51.173204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.173173 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:21:51.185434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.185408 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:21:51.189815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.186293 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:51.189815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.186339 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:51.189815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.186910 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:21:51.372336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.372278 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podStartSLOduration=7.372257951 podStartE2EDuration="7.372257951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:51.35185317 +0000 UTC m=+1465.516405710" watchObservedRunningTime="2026-11-02 15:21:51.372257951 +0000 UTC m=+1465.536810493" Nov 02 15:21:51.696582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.696459 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685"] Nov 02 15:21:51.697127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.697109 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7b3760e0-2421-4c88-a408-2200536074e1" containerName="revoke-keys" Nov 02 15:21:51.697188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.697128 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3760e0-2421-4c88-a408-2200536074e1" containerName="revoke-keys" Nov 02 15:21:51.697271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.697260 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3760e0-2421-4c88-a408-2200536074e1" containerName="revoke-keys" Nov 02 15:21:51.701147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.701016 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.757537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.754895 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:21:51.771373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.771336 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685"] Nov 02 15:21:51.798497 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.798450 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67t2h\" (UniqueName: \"kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.798784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.798760 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.798880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.798831 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.859541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.858811 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:21:51.900293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.900256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67t2h\" (UniqueName: \"kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.900469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.900435 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.900564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.900501 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.901478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.901444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.902954 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.902933 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:51.963153 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:51.963064 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67t2h\" (UniqueName: \"kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h\") pod \"maas-api-revoke-sub-e3336a14-hv685\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:52.015000 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:52.014963 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:52.015193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:52.015021 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:52.018236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:52.018204 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:52.470913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:52.469924 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685"] Nov 02 15:21:52.724318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:52.724231 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" event={"ID":"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b","Type":"ContainerStarted","Data":"fcba7cf9905f8ce08ba819d5623967d757002049353c1bb7a520c6a605fb7cea"} Nov 02 15:21:53.014417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:53.014386 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:53.014726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:53.014442 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:53.762377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:53.762195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" event={"ID":"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b","Type":"ContainerStarted","Data":"da085605be7f799c983feebe258417885a10c67fba2ce0a4274c284cbf3c2058"} Nov 02 15:21:53.884036 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:53.883971 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" podStartSLOduration=2.883952544 podStartE2EDuration="2.883952544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:53.882820335 +0000 UTC m=+1468.047372876" watchObservedRunningTime="2026-11-02 15:21:53.883952544 +0000 UTC m=+1468.048505087" Nov 02 15:21:53.983229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:53.982691 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8"] Nov 02 15:21:53.983229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:53.982885 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" containerID="cri-o://61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef" gracePeriod=30 Nov 02 15:21:54.014575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.014492 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:54.014732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.014595 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:54.223297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.223262 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg"] Nov 02 15:21:54.260660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.260627 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.289489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.288985 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg"] Nov 02 15:21:54.381545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.380243 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.381545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.380461 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.381545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.380583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-242b2\" (UniqueName: \"kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.481470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.481436 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.481755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.481732 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-242b2\" (UniqueName: \"kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.481930 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.481912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.482166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.482142 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.484463 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.484438 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.574708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.571268 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-242b2\" (UniqueName: \"kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2\") pod \"maas-api-revoke-sub-88f4867f-6tkkg\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.583370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.583337 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:54.775853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.775815 5091 generic.go:356] "Generic (PLEG): container finished" podID="3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" containerID="da085605be7f799c983feebe258417885a10c67fba2ce0a4274c284cbf3c2058" exitCode=0 Nov 02 15:21:54.776400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:54.775907 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" event={"ID":"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b","Type":"ContainerDied","Data":"da085605be7f799c983feebe258417885a10c67fba2ce0a4274c284cbf3c2058"} Nov 02 15:21:55.009029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:55.008999 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg"] Nov 02 15:21:55.012733 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:21:55.012707 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fd3e6a_8a60_48ba_a682_659300a49c21.slice/crio-9e588ebb2ea130ead2ee984eca0011b4eb7714502252f8279f1ee5b975e15f18 WatchSource:0}: Error finding container 9e588ebb2ea130ead2ee984eca0011b4eb7714502252f8279f1ee5b975e15f18: Status 404 returned error can't find the container with id 9e588ebb2ea130ead2ee984eca0011b4eb7714502252f8279f1ee5b975e15f18 Nov 02 15:21:55.014239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:55.014213 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:55.014390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:55.014346 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:55.785633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:55.785588 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" event={"ID":"f6fd3e6a-8a60-48ba-a682-659300a49c21","Type":"ContainerStarted","Data":"b5f82a28e416b0db97d4350db1a342ab0bd48d46b4ccb313c55a95f75cc824d5"} Nov 02 15:21:55.786134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:55.785659 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" event={"ID":"f6fd3e6a-8a60-48ba-a682-659300a49c21","Type":"ContainerStarted","Data":"9e588ebb2ea130ead2ee984eca0011b4eb7714502252f8279f1ee5b975e15f18"} Nov 02 15:21:55.950540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:55.926028 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" podStartSLOduration=1.926007539 podStartE2EDuration="1.926007539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:21:55.925618621 +0000 UTC m=+1470.090171163" watchObservedRunningTime="2026-11-02 15:21:55.926007539 +0000 UTC m=+1470.090560080" Nov 02 15:21:56.014543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.014265 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:56.014642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.014320 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:56.019455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.019437 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:56.108580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.108463 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca\") pod \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " Nov 02 15:21:56.108580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.108547 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca\") pod \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " Nov 02 15:21:56.108785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.108682 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h\" (UniqueName: \"kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h\") pod \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\" (UID: \"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b\") " Nov 02 15:21:56.108966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.108940 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca" pod "3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" (UID: "3a4a8178-03f9-49b2-80d4-f4b7ce758c0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:56.109251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.109236 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:56.110805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.110782 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca" pod "3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" (UID: "3a4a8178-03f9-49b2-80d4-f4b7ce758c0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:56.110897 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.110857 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h" pod "3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" (UID: "3a4a8178-03f9-49b2-80d4-f4b7ce758c0b"). InnerVolumeSpecName "kube-api-access-67t2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:56.211075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.211047 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:56.211180 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.211076 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67t2h\" (UniqueName: \"kubernetes.io/projected/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b-kube-api-access-67t2h\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:56.794594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.794553 5091 generic.go:356] "Generic (PLEG): container finished" podID="f6fd3e6a-8a60-48ba-a682-659300a49c21" containerID="b5f82a28e416b0db97d4350db1a342ab0bd48d46b4ccb313c55a95f75cc824d5" exitCode=0 Nov 02 15:21:56.795019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.794629 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" event={"ID":"f6fd3e6a-8a60-48ba-a682-659300a49c21","Type":"ContainerDied","Data":"b5f82a28e416b0db97d4350db1a342ab0bd48d46b4ccb313c55a95f75cc824d5"} Nov 02 15:21:56.796419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.796394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" event={"ID":"3a4a8178-03f9-49b2-80d4-f4b7ce758c0b","Type":"ContainerDied","Data":"fcba7cf9905f8ce08ba819d5623967d757002049353c1bb7a520c6a605fb7cea"} Nov 02 15:21:56.796419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.796417 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685" Nov 02 15:21:56.796593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:56.796439 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcba7cf9905f8ce08ba819d5623967d757002049353c1bb7a520c6a605fb7cea" Nov 02 15:21:57.014974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:57.014939 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 02 15:21:57.015173 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:57.014993 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 02 15:21:57.985284 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:21:57.984846 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:21:58.025339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.025308 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:58.142007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.141980 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:58.143380 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.143358 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:58.143934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.143914 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2\" (UniqueName: \"kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2\") pod \"f6fd3e6a-8a60-48ba-a682-659300a49c21\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " Nov 02 15:21:58.144031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.143963 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca\") pod \"f6fd3e6a-8a60-48ba-a682-659300a49c21\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " Nov 02 15:21:58.144087 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.144065 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca\") pod \"f6fd3e6a-8a60-48ba-a682-659300a49c21\" (UID: \"f6fd3e6a-8a60-48ba-a682-659300a49c21\") " Nov 02 15:21:58.148893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.148866 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca" pod "f6fd3e6a-8a60-48ba-a682-659300a49c21" (UID: "f6fd3e6a-8a60-48ba-a682-659300a49c21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:58.149191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.148874 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:21:58.165388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.151647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca" pod "f6fd3e6a-8a60-48ba-a682-659300a49c21" (UID: "f6fd3e6a-8a60-48ba-a682-659300a49c21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:21:58.165388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.156066 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2" pod "f6fd3e6a-8a60-48ba-a682-659300a49c21" (UID: "f6fd3e6a-8a60-48ba-a682-659300a49c21"). InnerVolumeSpecName "kube-api-access-242b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:58.245703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.245669 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-242b2\" (UniqueName: \"kubernetes.io/projected/f6fd3e6a-8a60-48ba-a682-659300a49c21-kube-api-access-242b2\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:58.245922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.245895 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:58.245922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.245918 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6fd3e6a-8a60-48ba-a682-659300a49c21-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:58.813246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.813213 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" Nov 02 15:21:58.813413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.813246 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg" event={"ID":"f6fd3e6a-8a60-48ba-a682-659300a49c21","Type":"ContainerDied","Data":"9e588ebb2ea130ead2ee984eca0011b4eb7714502252f8279f1ee5b975e15f18"} Nov 02 15:21:58.813413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:58.813295 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e588ebb2ea130ead2ee984eca0011b4eb7714502252f8279f1ee5b975e15f18" Nov 02 15:21:59.056031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.055992 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4"] Nov 02 15:21:59.059725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.059701 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6301bc7e-62pw4"] Nov 02 15:21:59.280970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.280945 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:21:59.359639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359567 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.359639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359612 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.359802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359654 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.359802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359740 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.359918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359825 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.359918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359872 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.360022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.360022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.359970 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.360128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360020 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.360128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360033 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.360128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360091 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert\") pod \"5dbc2167-c19a-43a3-be40-61607f5addf3\" (UID: \"5dbc2167-c19a-43a3-be40-61607f5addf3\") " Nov 02 15:21:59.360370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360347 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.360638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360587 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.360768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360683 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:21:59.360883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360857 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.360996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360890 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.360996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360907 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.360996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360921 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.360996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.360937 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dbc2167-c19a-43a3-be40-61607f5addf3-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.362263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.362238 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:21:59.362357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.362319 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:59.362511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.362494 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:21:59.362637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.362539 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9" pod "5dbc2167-c19a-43a3-be40-61607f5addf3" (UID: "5dbc2167-c19a-43a3-be40-61607f5addf3"). InnerVolumeSpecName "kube-api-access-4mhk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:21:59.462954 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.462922 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.462954 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.462952 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.463149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.462962 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.463149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.462971 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhk9\" (UniqueName: \"kubernetes.io/projected/5dbc2167-c19a-43a3-be40-61607f5addf3-kube-api-access-4mhk9\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:21:59.821201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.821162 5091 generic.go:356] "Generic (PLEG): container finished" podID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerID="61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef" exitCode=0 Nov 02 15:21:59.821399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.821209 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" Nov 02 15:21:59.821399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.821238 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" event={"ID":"5dbc2167-c19a-43a3-be40-61607f5addf3","Type":"ContainerDied","Data":"61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef"} Nov 02 15:21:59.821399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.821291 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8" event={"ID":"5dbc2167-c19a-43a3-be40-61607f5addf3","Type":"ContainerDied","Data":"5616ef04ac58807c424470ba88fd3556e2b7109891265d9c4bc7868deccb21fe"} Nov 02 15:21:59.821399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.821312 5091 scope.go:122] "RemoveContainer" containerID="61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef" Nov 02 15:21:59.838376 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.838356 5091 scope.go:122] "RemoveContainer" containerID="61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef" Nov 02 15:21:59.838673 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:21:59.838648 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef\": container with ID starting with 61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef not found: ID does not exist" containerID="61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef" Nov 02 15:21:59.838748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.838681 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef"} err="failed to get container status \"61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef\": rpc error: code = NotFound desc = could not find container \"61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef\": container with ID starting with 61f00911479d2812c750321860543f0086b481218a28f3807271f18b628423ef not found: ID does not exist" Nov 02 15:21:59.961330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.961289 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af7657c-bf64-424d-80fe-4a681a80707f" path="/var/lib/kubelet/pods/5af7657c-bf64-424d-80fe-4a681a80707f/volumes" Nov 02 15:21:59.981417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:21:59.981383 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8"] Nov 02 15:22:00.030534 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:00.030491 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-z59f8"] Nov 02 15:22:01.961335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:01.961300 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" path="/var/lib/kubelet/pods/5dbc2167-c19a-43a3-be40-61607f5addf3/volumes" Nov 02 15:22:03.647150 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647116 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6585-4b5b3c"] Nov 02 15:22:03.647787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647770 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f6fd3e6a-8a60-48ba-a682-659300a49c21" containerName="revoke-keys" Nov 02 15:22:03.647848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647789 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fd3e6a-8a60-48ba-a682-659300a49c21" containerName="revoke-keys" Nov 02 15:22:03.647848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647832 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" containerName="revoke-keys" Nov 02 15:22:03.647848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647837 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" containerName="revoke-keys" Nov 02 15:22:03.647946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647856 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" Nov 02 15:22:03.647946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647861 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" Nov 02 15:22:03.648010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647949 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" containerName="revoke-keys" Nov 02 15:22:03.648010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647959 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dbc2167-c19a-43a3-be40-61607f5addf3" containerName="istio-proxy" Nov 02 15:22:03.648010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.647966 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fd3e6a-8a60-48ba-a682-659300a49c21" containerName="revoke-keys" Nov 02 15:22:03.652876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.652858 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:03.703397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.703367 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6585-4b5b3c"] Nov 02 15:22:03.716881 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.716856 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj472\" (UniqueName: \"kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472\") pod \"test-curl-6585-4b5b3c\" (UID: \"1ba8ecd0-d838-4c70-8362-e1d8f430314b\") " pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:03.744110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.744087 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhrbh\"" Nov 02 15:22:03.818326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.818297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj472\" (UniqueName: \"kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472\") pod \"test-curl-6585-4b5b3c\" (UID: \"1ba8ecd0-d838-4c70-8362-e1d8f430314b\") " pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:03.876980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.876950 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj472\" (UniqueName: \"kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472\") pod \"test-curl-6585-4b5b3c\" (UID: \"1ba8ecd0-d838-4c70-8362-e1d8f430314b\") " pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:03.965188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:03.965112 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:04.351753 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:04.351678 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6585-4b5b3c"] Nov 02 15:22:04.355991 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:04.355963 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ba8ecd0_d838_4c70_8362_e1d8f430314b.slice/crio-2972dc1dcb56d519f2bb53c145c74d01b6108c451ca4b9cb6d9f35cef272db6f WatchSource:0}: Error finding container 2972dc1dcb56d519f2bb53c145c74d01b6108c451ca4b9cb6d9f35cef272db6f: Status 404 returned error can't find the container with id 2972dc1dcb56d519f2bb53c145c74d01b6108c451ca4b9cb6d9f35cef272db6f Nov 02 15:22:04.358164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:04.358142 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:22:04.863831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:04.863798 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6585-4b5b3c" event={"ID":"1ba8ecd0-d838-4c70-8362-e1d8f430314b","Type":"ContainerStarted","Data":"2972dc1dcb56d519f2bb53c145c74d01b6108c451ca4b9cb6d9f35cef272db6f"} Nov 02 15:22:04.968601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:04.968572 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9"] Nov 02 15:22:04.973694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:04.973676 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.028504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.028475 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9"] Nov 02 15:22:05.070372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.070334 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:05.070372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.070334 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:05.137801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.137770 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.138048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.137911 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghkf\" (UniqueName: \"kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.138048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.137951 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.239607 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.239574 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.239795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.239642 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghkf\" (UniqueName: \"kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.239795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.239751 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.240383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.240360 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.241940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.241922 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.297929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.297898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghkf\" (UniqueName: \"kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf\") pod \"maas-api-revoke-sub-eb8400db-69bn9\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.588461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.588385 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:05.971992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:05.971959 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9"] Nov 02 15:22:05.976175 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:05.976129 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f97404_f95f_4e66_b2b0_deb4c8ba7e7a.slice/crio-ff3f7ecd8cb18cbc6901272aef13cee59977984df31e38efe06bef2f56f347d2 WatchSource:0}: Error finding container ff3f7ecd8cb18cbc6901272aef13cee59977984df31e38efe06bef2f56f347d2: Status 404 returned error can't find the container with id ff3f7ecd8cb18cbc6901272aef13cee59977984df31e38efe06bef2f56f347d2 Nov 02 15:22:06.886077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:06.886030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" event={"ID":"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a","Type":"ContainerStarted","Data":"39d703ae6d780f7323fbe8dd34e4072810910f38ced3624e48c485f1a62a6c56"} Nov 02 15:22:06.886256 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:06.886144 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" event={"ID":"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a","Type":"ContainerStarted","Data":"ff3f7ecd8cb18cbc6901272aef13cee59977984df31e38efe06bef2f56f347d2"} Nov 02 15:22:07.001120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:07.001067 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" podStartSLOduration=3.001054251 podStartE2EDuration="3.001054251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:06.996689757 +0000 UTC m=+1481.161242298" watchObservedRunningTime="2026-11-02 15:22:07.001054251 +0000 UTC m=+1481.165606803" Nov 02 15:22:07.895088 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:07.895045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6585-4b5b3c" event={"ID":"1ba8ecd0-d838-4c70-8362-e1d8f430314b","Type":"ContainerStarted","Data":"33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7"} Nov 02 15:22:07.896936 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:07.896908 5091 generic.go:356] "Generic (PLEG): container finished" podID="b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" containerID="39d703ae6d780f7323fbe8dd34e4072810910f38ced3624e48c485f1a62a6c56" exitCode=0 Nov 02 15:22:07.897064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:07.896995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" event={"ID":"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a","Type":"ContainerDied","Data":"39d703ae6d780f7323fbe8dd34e4072810910f38ced3624e48c485f1a62a6c56"} Nov 02 15:22:08.005580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:08.005530 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6585-4b5b3c" podStartSLOduration=2.145894711 podStartE2EDuration="5.005503068s" totalImagesPullingTime="2.859608357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:08.004001083 +0000 UTC m=+1482.168553616" watchObservedRunningTime="2026-11-02 15:22:08.005503068 +0000 UTC m=+1482.170055609" Nov 02 15:22:09.072851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.072826 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:09.110450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.110424 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf\" (UniqueName: \"kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf\") pod \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " Nov 02 15:22:09.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.110536 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca\") pod \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " Nov 02 15:22:09.110597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.110560 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca\") pod \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\" (UID: \"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a\") " Nov 02 15:22:09.110968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.110876 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca" pod "b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" (UID: "b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:09.111095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.111039 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:09.112700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.112678 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca" pod "b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" (UID: "b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:09.113053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.113034 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf" pod "b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" (UID: "b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a"). InnerVolumeSpecName "kube-api-access-bghkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:09.212473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.212394 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghkf\" (UniqueName: \"kubernetes.io/projected/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-kube-api-access-bghkf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:09.212473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.212425 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:09.524359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.524282 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6585-4b5b3c"] Nov 02 15:22:09.529701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.529672 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6585-4b5b3c"] Nov 02 15:22:09.682860 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.682828 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6585-4b4e78"] Nov 02 15:22:09.683476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.683460 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" containerName="revoke-keys" Nov 02 15:22:09.683562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.683478 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" containerName="revoke-keys" Nov 02 15:22:09.683562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.683507 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" containerName="test-curl-6585-4b5b3c" Nov 02 15:22:09.683562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.683524 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" containerName="test-curl-6585-4b5b3c" Nov 02 15:22:09.683679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.683626 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" containerName="test-curl-6585-4b5b3c" Nov 02 15:22:09.683679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.683642 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" containerName="revoke-keys" Nov 02 15:22:09.687296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.687276 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:09.718926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.718899 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpnr\" (UniqueName: \"kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr\") pod \"test-curl-6585-4b4e78\" (UID: \"d32fb971-a0d9-4537-b39a-07e906552562\") " pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:09.738563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.737710 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6585-4b4e78"] Nov 02 15:22:09.820266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.820191 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpnr\" (UniqueName: \"kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr\") pod \"test-curl-6585-4b4e78\" (UID: \"d32fb971-a0d9-4537-b39a-07e906552562\") " pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:09.878367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.878338 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpnr\" (UniqueName: \"kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr\") pod \"test-curl-6585-4b4e78\" (UID: \"d32fb971-a0d9-4537-b39a-07e906552562\") " pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:09.912221 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.912193 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" Nov 02 15:22:09.912371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.912221 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9" event={"ID":"b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a","Type":"ContainerDied","Data":"ff3f7ecd8cb18cbc6901272aef13cee59977984df31e38efe06bef2f56f347d2"} Nov 02 15:22:09.912371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.912275 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff3f7ecd8cb18cbc6901272aef13cee59977984df31e38efe06bef2f56f347d2" Nov 02 15:22:09.912473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:09.912447 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6585-4b5b3c" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" containerName="test-curl-6585-4b5b3c" containerID="cri-o://33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7" gracePeriod=2 Nov 02 15:22:09.960963 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:09.960932 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6585-4b5b3c\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:10.000905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:10.000879 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:10.009384 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:10.009351 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6585-4b5b3c\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:10.383016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:10.382984 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6585-4b4e78"] Nov 02 15:22:10.387044 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:10.387003 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd32fb971_a0d9_4537_b39a_07e906552562.slice/crio-c4e11721f832f42e50bdcc252d507ec9977a19171dff594d1f174df84f985ff7 WatchSource:0}: Error finding container c4e11721f832f42e50bdcc252d507ec9977a19171dff594d1f174df84f985ff7: Status 404 returned error can't find the container with id c4e11721f832f42e50bdcc252d507ec9977a19171dff594d1f174df84f985ff7 Nov 02 15:22:10.923090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:10.923052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6585-4b4e78" event={"ID":"d32fb971-a0d9-4537-b39a-07e906552562","Type":"ContainerStarted","Data":"2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5"} Nov 02 15:22:10.923298 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:10.923104 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6585-4b4e78" event={"ID":"d32fb971-a0d9-4537-b39a-07e906552562","Type":"ContainerStarted","Data":"c4e11721f832f42e50bdcc252d507ec9977a19171dff594d1f174df84f985ff7"} Nov 02 15:22:11.035631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:11.035562 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6585-4b4e78" podStartSLOduration=2.035543798 podStartE2EDuration="2.035543798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:11.033082494 +0000 UTC m=+1485.197635035" watchObservedRunningTime="2026-11-02 15:22:11.035543798 +0000 UTC m=+1485.200096344" Nov 02 15:22:12.163437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.163407 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:12.256777 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.256707 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472\" (UniqueName: \"kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472\") pod \"1ba8ecd0-d838-4c70-8362-e1d8f430314b\" (UID: \"1ba8ecd0-d838-4c70-8362-e1d8f430314b\") " Nov 02 15:22:12.258750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.258719 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472" pod "1ba8ecd0-d838-4c70-8362-e1d8f430314b" (UID: "1ba8ecd0-d838-4c70-8362-e1d8f430314b"). InnerVolumeSpecName "kube-api-access-rj472". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:12.358335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.358300 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj472\" (UniqueName: \"kubernetes.io/projected/1ba8ecd0-d838-4c70-8362-e1d8f430314b-kube-api-access-rj472\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:12.588987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.588890 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6585-4b4e78"] Nov 02 15:22:12.597258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.597226 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6585-4b4e78"] Nov 02 15:22:12.941139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.941100 5091 generic.go:356] "Generic (PLEG): container finished" podID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" containerID="33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7" exitCode=137 Nov 02 15:22:12.941328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.941202 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:12.941328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.941245 5091 scope.go:122] "RemoveContainer" containerID="33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7" Nov 02 15:22:12.941604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.941577 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6585-4b4e78" podUID="d32fb971-a0d9-4537-b39a-07e906552562" containerName="test-curl-6585-4b4e78" containerID="cri-o://2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5" gracePeriod=2 Nov 02 15:22:12.971054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.971032 5091 scope.go:122] "RemoveContainer" containerID="33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7" Nov 02 15:22:12.971357 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:12.971328 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7\": container with ID starting with 33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7 not found: ID does not exist" containerID="33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7" Nov 02 15:22:12.971413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:12.971367 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7"} err="failed to get container status \"33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7\": rpc error: code = NotFound desc = could not find container \"33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7\": container with ID starting with 33196cd44b5dcc5337b9b9dcabeae7d4e7f1b361d2fd177e26baa30780f060a7 not found: ID does not exist" Nov 02 15:22:12.990095 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:12.990063 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6585-4b4e78\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="d32fb971-a0d9-4537-b39a-07e906552562" pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:13.038586 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:13.038548 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6585-4b4e78\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="d32fb971-a0d9-4537-b39a-07e906552562" pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:13.086996 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:13.086959 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6585-4b5b3c\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" pod="opendatahub/test-curl-6585-4b5b3c" Nov 02 15:22:14.005982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:14.005359 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba8ecd0-d838-4c70-8362-e1d8f430314b" path="/var/lib/kubelet/pods/1ba8ecd0-d838-4c70-8362-e1d8f430314b/volumes" Nov 02 15:22:15.199277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.199251 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:15.314334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.314260 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr\" (UniqueName: \"kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr\") pod \"d32fb971-a0d9-4537-b39a-07e906552562\" (UID: \"d32fb971-a0d9-4537-b39a-07e906552562\") " Nov 02 15:22:15.316433 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.316400 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr" pod "d32fb971-a0d9-4537-b39a-07e906552562" (UID: "d32fb971-a0d9-4537-b39a-07e906552562"). InnerVolumeSpecName "kube-api-access-9bpnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:15.415878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.415848 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpnr\" (UniqueName: \"kubernetes.io/projected/d32fb971-a0d9-4537-b39a-07e906552562-kube-api-access-9bpnr\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:15.966512 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.966470 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32fb971-a0d9-4537-b39a-07e906552562" path="/var/lib/kubelet/pods/d32fb971-a0d9-4537-b39a-07e906552562/volumes" Nov 02 15:22:15.970935 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.970900 5091 generic.go:356] "Generic (PLEG): container finished" podID="d32fb971-a0d9-4537-b39a-07e906552562" containerID="2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5" exitCode=137 Nov 02 15:22:15.971118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.970998 5091 scope.go:122] "RemoveContainer" containerID="2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5" Nov 02 15:22:15.971279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.971261 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6585-4b4e78" Nov 02 15:22:15.984303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.984281 5091 scope.go:122] "RemoveContainer" containerID="2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5" Nov 02 15:22:15.984665 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:15.984636 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5\": container with ID starting with 2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5 not found: ID does not exist" containerID="2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5" Nov 02 15:22:15.984730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:15.984676 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5"} err="failed to get container status \"2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5\": rpc error: code = NotFound desc = could not find container \"2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5\": container with ID starting with 2970097f89ab5836f2c74a9401869e990d80ed6009a04a91e4986a1a3f6873d5 not found: ID does not exist" Nov 02 15:22:18.706583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.706551 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5"] Nov 02 15:22:18.707189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.707174 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d32fb971-a0d9-4537-b39a-07e906552562" containerName="test-curl-6585-4b4e78" Nov 02 15:22:18.707235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.707190 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32fb971-a0d9-4537-b39a-07e906552562" containerName="test-curl-6585-4b4e78" Nov 02 15:22:18.707293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.707284 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32fb971-a0d9-4537-b39a-07e906552562" containerName="test-curl-6585-4b4e78" Nov 02 15:22:18.712166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.712144 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.766444 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.766415 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5"] Nov 02 15:22:18.777080 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.777046 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.777243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.777183 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwd6k\" (UniqueName: \"kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.777307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.777263 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.813680 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.813654 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:18.813809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.813716 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:18.878756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.878725 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.878927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.878787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwd6k\" (UniqueName: \"kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.878927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.878828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.879483 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.879457 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.881178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.881157 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:18.939199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:18.939161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwd6k\" (UniqueName: \"kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k\") pod \"maas-api-revoke-sub-a0021fc6-lrdr5\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:19.026042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:19.025957 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:19.403843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:19.403813 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5"] Nov 02 15:22:19.407446 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:19.407419 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c WatchSource:0}: Error finding container f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c: Status 404 returned error can't find the container with id f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c Nov 02 15:22:20.002419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.002337 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" event={"ID":"5e938984-35f5-4011-a10b-b0c8ae2d72b7","Type":"ContainerStarted","Data":"3fca94fd4ea3128b9f4c0df9dde3ae1f77cd7c38683add82a3f6663e1baed6da"} Nov 02 15:22:20.002419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.002394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" event={"ID":"5e938984-35f5-4011-a10b-b0c8ae2d72b7","Type":"ContainerStarted","Data":"f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c"} Nov 02 15:22:20.113380 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.113328 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" podStartSLOduration=2.113315175 podStartE2EDuration="2.113315175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:20.112833783 +0000 UTC m=+1494.277386324" watchObservedRunningTime="2026-11-02 15:22:20.113315175 +0000 UTC m=+1494.277867715" Nov 02 15:22:20.517292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.517253 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q"] Nov 02 15:22:20.521844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.521818 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.580902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.580864 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q"] Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602127 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602290 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602366 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602402 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrjj\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602542 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602713 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.605575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.602921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.612062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.612036 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bd9398b4-openshift-default-dockercfg-zhgsn\"" Nov 02 15:22:20.704160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704132 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704233 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704275 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704397 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704476 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704579 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704614 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704644 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrjj\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704654 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704865 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.704979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.704956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.705127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.705104 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.706765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.706733 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.706992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.706974 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.780540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.775269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrjj\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.780540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.775769 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token\") pod \"e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:20.864596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:20.863571 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:21.145602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.144167 5091 generic.go:356] "Generic (PLEG): container finished" podID="5e938984-35f5-4011-a10b-b0c8ae2d72b7" containerID="3fca94fd4ea3128b9f4c0df9dde3ae1f77cd7c38683add82a3f6663e1baed6da" exitCode=0 Nov 02 15:22:21.145602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.144555 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" event={"ID":"5e938984-35f5-4011-a10b-b0c8ae2d72b7","Type":"ContainerDied","Data":"3fca94fd4ea3128b9f4c0df9dde3ae1f77cd7c38683add82a3f6663e1baed6da"} Nov 02 15:22:21.744542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.725698 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q"] Nov 02 15:22:21.805687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.805576 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:21.805687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.805664 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:21.805882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.805715 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:21.858548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.854029 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522"] Nov 02 15:22:21.884381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:21.884350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.036427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.036334 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a758f7bf-dockercfg-252kc\"" Nov 02 15:22:22.036879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.036857 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a758f7bf\"" Nov 02 15:22:22.119556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.112187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmgss\" (UniqueName: \"kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss\") pod \"payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.119556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.112249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.213874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.213830 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmgss\" (UniqueName: \"kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss\") pod \"payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.255338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.255301 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.255809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.252982 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522"] Nov 02 15:22:22.265541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.256616 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.345539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.335663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmgss\" (UniqueName: \"kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss\") pod \"payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.388541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.387638 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" event={"ID":"c1f21248-035b-4e39-9691-73f0b5a4e373","Type":"ContainerStarted","Data":"9e1f68b745e1311b8636f68c11459e5da4697ebcf7920f6b6b8b3fbf1d8026ae"} Nov 02 15:22:22.414969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.414921 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:22.443539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.440240 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:22:22.443539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.440324 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.443539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.440191 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:22.542304 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.542272 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck"] Nov 02 15:22:22.670540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.664900 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.714055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.714018 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck"] Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740228 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740355 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740395 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740440 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740476 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740572 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2dp\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740654 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.744658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.740848 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.761900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.761864 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-06a527d4-gw-openshift-default-dockercfg-ghhgc\"" Nov 02 15:22:22.865848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.843432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.866329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.866304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:22.866477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.866458 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.103020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.879412 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.126903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.843784 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.127251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.879714 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.127490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.883609 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.128191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.982167 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.128335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:22.993752 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.128776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.041661 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.129155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.103264 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.129399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.129364 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.129589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.129572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.129824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.129796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.168621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.163058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2dp\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.168621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.163186 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.168621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.103713 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.168621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.153959 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.190258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.162903 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.204990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.203981 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.290545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.249025 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2dp\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.329296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.329215 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo\") pod \"e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.456641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.455781 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:22:23.460597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.457171 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:22:23.521541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.517998 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:23.531244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.531201 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:23.574952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.566890 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 02 15:22:23.864548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:23.855725 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.054636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.037989 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:24.125844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.125786 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.150915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.135566 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.195143 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.149813 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.196202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.179104 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.325495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.196633 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:24.358588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.357631 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.468554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.467042 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.608592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.589653 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.608592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.591484 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" event={"ID":"c1f21248-035b-4e39-9691-73f0b5a4e373","Type":"ContainerStarted","Data":"2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18"} Nov 02 15:22:24.608592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.591597 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:22:24.608592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.591638 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:22:24.608592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.591673 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:22:24.715646 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.715590 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:24.951558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.925141 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:22:24.951558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.925489 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:22:24.988816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:24.963031 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:25.052297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.049077 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:25.093564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.092969 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:22:25.215606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.186551 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.215606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.203144 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:25.239724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.239675 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.246759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.246720 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:22:25.253244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.253210 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:22:25.267226 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.267197 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:22:25.313053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.313012 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca\") pod \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " Nov 02 15:22:25.359257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.353727 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca" pod "5e938984-35f5-4011-a10b-b0c8ae2d72b7" (UID: "5e938984-35f5-4011-a10b-b0c8ae2d72b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:25.398418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.363148 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca\") pod \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " Nov 02 15:22:25.399398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.399377 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k\" (UniqueName: \"kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k\") pod \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\" (UID: \"5e938984-35f5-4011-a10b-b0c8ae2d72b7\") " Nov 02 15:22:25.400470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.400442 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:25.413699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.406465 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 15:22:25.415005 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.409606 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:25.436543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.415184 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.487597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.480599 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.491539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.489657 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podStartSLOduration=5.489635289 podStartE2EDuration="5.489635289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:24.741313697 +0000 UTC m=+1498.905866237" watchObservedRunningTime="2026-11-02 15:22:25.489635289 +0000 UTC m=+1499.654187830" Nov 02 15:22:25.589693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.544197 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:22:25.589693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.544267 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.653583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.637218 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:22:25.753608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.726383 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 15:22:25.753608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.727234 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:22:25.793455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.793418 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca" pod "5e938984-35f5-4011-a10b-b0c8ae2d72b7" (UID: "5e938984-35f5-4011-a10b-b0c8ae2d72b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:25.828931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.821906 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:22:25.928673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.928629 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:25.929237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.929208 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/health\": context deadline exceeded" Nov 02 15:22:25.944691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:25.944656 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k" pod "5e938984-35f5-4011-a10b-b0c8ae2d72b7" (UID: "5e938984-35f5-4011-a10b-b0c8ae2d72b7"). InnerVolumeSpecName "kube-api-access-gwd6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:26.032596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.005716 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" Nov 02 15:22:26.333610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.325076 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 15:22:26.399868 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.399822 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:22:26.414629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.414584 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 15:22:26.635861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.474937 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwd6k\" (UniqueName: \"kubernetes.io/projected/5e938984-35f5-4011-a10b-b0c8ae2d72b7-kube-api-access-gwd6k\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:26.635861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.607242 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e938984-35f5-4011-a10b-b0c8ae2d72b7-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:26.635861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.589474 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Nov 02 15:22:26.635861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.495615 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.653538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.647633 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:22:26.653538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.652541 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.657549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.654075 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:22:26.670547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.668357 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.775567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.754652 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:26.904595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.904483 5091 scope.go:122] "RemoveContainer" containerID="ffa64e352357f35721b28314f9da5babacd6d1fe12b4dca0b92c83eed6dabb98" Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908642 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5" event={"ID":"5e938984-35f5-4011-a10b-b0c8ae2d72b7","Type":"ContainerDied","Data":"f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c"} Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908684 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c" Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908704 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908749 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522"] Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908843 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908888 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.908932 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:22:27.047572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:26.930689 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck"] Nov 02 15:22:27.432398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.432351 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:22:27.458818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.454409 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:27.458818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.454595 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:27.458818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.454706 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:27.458818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.456467 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Nov 02 15:22:27.514537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.510673 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.560499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.560427 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:22:27.628198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.626026 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:22:27.628198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.627678 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 15:22:27.651994 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.649070 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.669656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.669613 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:27.670279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.670247 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:22:27.670871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.670850 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:22:27.686748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.686690 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded" Nov 02 15:22:27.687302 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.687258 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": context deadline exceeded" Nov 02 15:22:27.698075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.687822 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:22:27.788961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:27.788861 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 02 15:22:28.206568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.206538 5091 scope.go:122] "RemoveContainer" containerID="ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6" Nov 02 15:22:28.376907 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:28.376867 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:22:28.456775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.435691 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:28.474956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.474915 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:22:28.478400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.478369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" event={"ID":"c39978d8-1bfa-4bb4-a8fb-394c9b277d51","Type":"ContainerStarted","Data":"b243604d350086658f1ef3bf0a5f585a3016049d7b62710d2b170cca5fe9fd0d"} Nov 02 15:22:28.479219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.478956 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:22:28.479616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479595 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479646 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479678 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479709 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479723 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" event={"ID":"166421cc-2839-41e7-8ac8-223db1bdc0de","Type":"ContainerStarted","Data":"3d1ad2ed3814f68573d88c6d7e3536cba1768bd38c72a61a4c89b394d2969366"} Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479771 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479801 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479828 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479857 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:22:28.479973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.479885 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:22:28.481445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.481419 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:22:28.501165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.501142 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:22:28.511943 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:28.511821 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6\": container with ID starting with ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6 not found: ID does not exist" containerID="ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6" Nov 02 15:22:28.511943 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:28.511894 5091 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6\": container with ID starting with ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6 not found: ID does not exist" Nov 02 15:22:28.514929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:28.514905 5091 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6\": container with ID starting with ae9a8669de8f4c0f1a54333a8d2fac57203db70902433c688524da2f96a5f2e6 not found: ID does not exist" Nov 02 15:22:29.239569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:29.239492 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" event={"ID":"c39978d8-1bfa-4bb4-a8fb-394c9b277d51","Type":"ContainerStarted","Data":"91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff"} Nov 02 15:22:29.239775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:29.239598 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:29.241532 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:29.241471 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" event={"ID":"166421cc-2839-41e7-8ac8-223db1bdc0de","Type":"ContainerStarted","Data":"b78050d8245551571d6eda65ea53e3386ec49d1954cb8e2194fe9598c12bcf08"} Nov 02 15:22:29.521161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:29.521014 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:29.527661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:29.527633 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:30.206007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:30.205944 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" podStartSLOduration=9.205929267 podStartE2EDuration="9.205929267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:30.203762005 +0000 UTC m=+1504.368314547" watchObservedRunningTime="2026-11-02 15:22:30.205929267 +0000 UTC m=+1504.370481807" Nov 02 15:22:30.255243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:30.251465 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:30.255243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:30.253123 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:22:30.460596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:30.458493 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podStartSLOduration=8.458472924 podStartE2EDuration="8.458472924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:30.330613241 +0000 UTC m=+1504.495165795" watchObservedRunningTime="2026-11-02 15:22:30.458472924 +0000 UTC m=+1504.623025464" Nov 02 15:22:31.484140 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.484107 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5"] Nov 02 15:22:31.484821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.484803 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5e938984-35f5-4011-a10b-b0c8ae2d72b7" containerName="revoke-keys" Nov 02 15:22:31.484821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.484822 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e938984-35f5-4011-a10b-b0c8ae2d72b7" containerName="revoke-keys" Nov 02 15:22:31.484987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.484976 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e938984-35f5-4011-a10b-b0c8ae2d72b7" containerName="revoke-keys" Nov 02 15:22:31.490858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.490739 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.545973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.545935 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5"] Nov 02 15:22:31.586431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.586392 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:31.586658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.586503 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:31.592830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.592805 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.593039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.593021 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.593206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.593185 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd6qv\" (UniqueName: \"kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.695540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.694688 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd6qv\" (UniqueName: \"kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.695540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.694771 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.695540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.694912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.701855 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.697023 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.702205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.702176 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.768584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.766444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd6qv\" (UniqueName: \"kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv\") pod \"maas-api-revoke-sub-57d91221-wj8m5\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:31.807484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:31.807448 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:32.191973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:32.191938 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5"] Nov 02 15:22:32.195593 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:32.195567 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb6cbc1_a7ca_4d13_a26a_61e1c8cc3744.slice/crio-d9c50c800df4b4248fba10594ab85f7e0a75884a4493952bc408a819061e63e4 WatchSource:0}: Error finding container d9c50c800df4b4248fba10594ab85f7e0a75884a4493952bc408a819061e63e4: Status 404 returned error can't find the container with id d9c50c800df4b4248fba10594ab85f7e0a75884a4493952bc408a819061e63e4 Nov 02 15:22:32.271939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:32.271904 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" event={"ID":"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744","Type":"ContainerStarted","Data":"d9c50c800df4b4248fba10594ab85f7e0a75884a4493952bc408a819061e63e4"} Nov 02 15:22:33.280851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:33.280816 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" containerID="51ebefdb258471099a70f19dc6d9ffe736a67e051c0d288118e8d9cb6e9a6321" exitCode=0 Nov 02 15:22:33.281233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:33.280859 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" event={"ID":"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744","Type":"ContainerDied","Data":"51ebefdb258471099a70f19dc6d9ffe736a67e051c0d288118e8d9cb6e9a6321"} Nov 02 15:22:34.480561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.480506 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:34.635752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.635722 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv\" (UniqueName: \"kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv\") pod \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " Nov 02 15:22:34.635942 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.635926 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca\") pod \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " Nov 02 15:22:34.636034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.636022 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca\") pod \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\" (UID: \"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744\") " Nov 02 15:22:34.636287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.636264 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca" pod "2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" (UID: "2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:34.636797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.636772 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:34.638093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.638066 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv" pod "2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" (UID: "2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744"). InnerVolumeSpecName "kube-api-access-cd6qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:34.638231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.638082 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca" pod "2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" (UID: "2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:34.738297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.738263 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd6qv\" (UniqueName: \"kubernetes.io/projected/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-kube-api-access-cd6qv\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:34.738297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:34.738292 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:35.298408 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.298378 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" Nov 02 15:22:35.298583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.298410 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5" event={"ID":"2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744","Type":"ContainerDied","Data":"d9c50c800df4b4248fba10594ab85f7e0a75884a4493952bc408a819061e63e4"} Nov 02 15:22:35.298583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.298466 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c50c800df4b4248fba10594ab85f7e0a75884a4493952bc408a819061e63e4" Nov 02 15:22:35.788641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.788607 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc"] Nov 02 15:22:35.789257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.789238 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" containerName="revoke-keys" Nov 02 15:22:35.789306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.789266 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" containerName="revoke-keys" Nov 02 15:22:35.789409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.789398 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" containerName="revoke-keys" Nov 02 15:22:35.793575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.793554 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:35.844347 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.844323 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:35.851556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.849501 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc"] Nov 02 15:22:35.892596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.890132 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:35.974566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.971325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrt7m\" (UniqueName: \"kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:35.974566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.972150 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:35.974566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:35.972257 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.073783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.073676 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrt7m\" (UniqueName: \"kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.073949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.073936 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.074033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.074015 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.074946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.074918 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.076697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.076677 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.138574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.138533 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrt7m\" (UniqueName: \"kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m\") pod \"maas-api-revoke-sub-ece7c605-ldqjc\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.197452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.197419 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9"] Nov 02 15:22:36.203048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.203011 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.255002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.254960 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-06a527d4\"" Nov 02 15:22:36.263700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.263654 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9"] Nov 02 15:22:36.285075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.284982 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgd4s\" (UniqueName: \"kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s\") pod \"payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.285283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.285242 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.387278 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.387237 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgd4s\" (UniqueName: \"kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s\") pod \"payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.387507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.387412 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.388507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.388473 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.408551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.408478 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:36.535332 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.522536 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgd4s\" (UniqueName: \"kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s\") pod \"payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.759410 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:36.759314 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:22:36.820455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.820416 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:36.958658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:36.958622 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc"] Nov 02 15:22:36.963290 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:36.963261 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c86771_f35b_4a31_bcd6_16c06c9a6b8f.slice/crio-06c94a39cffaf175e461551ed5191a4ac9b769a647c385db2eed6e646baa93bc WatchSource:0}: Error finding container 06c94a39cffaf175e461551ed5191a4ac9b769a647c385db2eed6e646baa93bc: Status 404 returned error can't find the container with id 06c94a39cffaf175e461551ed5191a4ac9b769a647c385db2eed6e646baa93bc Nov 02 15:22:37.198629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:37.198598 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9"] Nov 02 15:22:37.202544 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:37.202499 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98ef18e_dc40_488b_98c2_5288176b6975.slice/crio-c4f4c8b04eb1acf0d7ae0d5deed4550ab0fb0722d99e570e1351433939b5fdee WatchSource:0}: Error finding container c4f4c8b04eb1acf0d7ae0d5deed4550ab0fb0722d99e570e1351433939b5fdee: Status 404 returned error can't find the container with id c4f4c8b04eb1acf0d7ae0d5deed4550ab0fb0722d99e570e1351433939b5fdee Nov 02 15:22:37.318071 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:37.318032 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" event={"ID":"17c86771-f35b-4a31-bcd6-16c06c9a6b8f","Type":"ContainerStarted","Data":"06c94a39cffaf175e461551ed5191a4ac9b769a647c385db2eed6e646baa93bc"} Nov 02 15:22:37.319755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:37.319726 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" event={"ID":"c98ef18e-dc40-488b-98c2-5288176b6975","Type":"ContainerStarted","Data":"43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9"} Nov 02 15:22:37.319887 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:37.319774 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" event={"ID":"c98ef18e-dc40-488b-98c2-5288176b6975","Type":"ContainerStarted","Data":"c4f4c8b04eb1acf0d7ae0d5deed4550ab0fb0722d99e570e1351433939b5fdee"} Nov 02 15:22:37.319887 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:37.319796 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:37.432540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:37.432427 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" podStartSLOduration=1.432410052 podStartE2EDuration="1.432410052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:37.430340992 +0000 UTC m=+1511.594893532" watchObservedRunningTime="2026-11-02 15:22:37.432410052 +0000 UTC m=+1511.596962593" Nov 02 15:22:38.330047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:38.329959 5091 generic.go:356] "Generic (PLEG): container finished" podID="17c86771-f35b-4a31-bcd6-16c06c9a6b8f" containerID="8206974f2c36f24f6a3648df3655bec15539f9b7d2c606f3b207a8f40129f146" exitCode=0 Nov 02 15:22:38.330427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:38.330045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" event={"ID":"17c86771-f35b-4a31-bcd6-16c06c9a6b8f","Type":"ContainerDied","Data":"8206974f2c36f24f6a3648df3655bec15539f9b7d2c606f3b207a8f40129f146"} Nov 02 15:22:38.594663 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:38.594582 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:22:39.307222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.307184 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9"] Nov 02 15:22:39.307488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.307452 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" containerID="cri-o://d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca" gracePeriod=30 Nov 02 15:22:39.560300 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.560240 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:39.666245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.666213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca\") pod \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " Nov 02 15:22:39.666462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.666339 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m\" (UniqueName: \"kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m\") pod \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " Nov 02 15:22:39.666462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.666449 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca\") pod \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\" (UID: \"17c86771-f35b-4a31-bcd6-16c06c9a6b8f\") " Nov 02 15:22:39.666686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.666656 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca" pod "17c86771-f35b-4a31-bcd6-16c06c9a6b8f" (UID: "17c86771-f35b-4a31-bcd6-16c06c9a6b8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:39.667214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.667192 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:39.668549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.668486 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca" pod "17c86771-f35b-4a31-bcd6-16c06c9a6b8f" (UID: "17c86771-f35b-4a31-bcd6-16c06c9a6b8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:39.668666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.668644 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m" pod "17c86771-f35b-4a31-bcd6-16c06c9a6b8f" (UID: "17c86771-f35b-4a31-bcd6-16c06c9a6b8f"). InnerVolumeSpecName "kube-api-access-hrt7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:39.768408 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.768368 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrt7m\" (UniqueName: \"kubernetes.io/projected/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-kube-api-access-hrt7m\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:39.768408 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:39.768401 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17c86771-f35b-4a31-bcd6-16c06c9a6b8f-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:40.254704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:40.254623 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:22:40.359248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:40.359217 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" Nov 02 15:22:40.359413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:40.359242 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc" event={"ID":"17c86771-f35b-4a31-bcd6-16c06c9a6b8f","Type":"ContainerDied","Data":"06c94a39cffaf175e461551ed5191a4ac9b769a647c385db2eed6e646baa93bc"} Nov 02 15:22:40.359413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:40.359300 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06c94a39cffaf175e461551ed5191a4ac9b769a647c385db2eed6e646baa93bc" Nov 02 15:22:42.356850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.356813 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk"] Nov 02 15:22:42.357581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.357563 5091 cpu_manager.go:414] "containerMap: removing container" podUID="17c86771-f35b-4a31-bcd6-16c06c9a6b8f" containerName="revoke-keys" Nov 02 15:22:42.357581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.357581 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c86771-f35b-4a31-bcd6-16c06c9a6b8f" containerName="revoke-keys" Nov 02 15:22:42.357723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.357714 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c86771-f35b-4a31-bcd6-16c06c9a6b8f" containerName="revoke-keys" Nov 02 15:22:42.366922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.366893 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.417403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.417370 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk"] Nov 02 15:22:42.457957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.457927 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:42.458204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.458191 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:42.513116 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.513076 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.513364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.513273 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.513436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.513385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kj9\" (UniqueName: \"kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.614944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.614917 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.615104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.614981 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kj9\" (UniqueName: \"kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.615104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.615096 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.615747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.615717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.617508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.617490 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.674137 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.674102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kj9\" (UniqueName: \"kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9\") pod \"maas-api-revoke-sub-2698f4a9-nqdpk\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:42.682553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:42.682500 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:43.062901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:43.062823 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk"] Nov 02 15:22:43.066768 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:43.066739 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a33e49_9b35_47a8_a7f6_08d198062c1a.slice/crio-575cba3d0c5cc232f7fb3aa2ce7a7130e08c26c9723e328133646d0c0fa61aeb WatchSource:0}: Error finding container 575cba3d0c5cc232f7fb3aa2ce7a7130e08c26c9723e328133646d0c0fa61aeb: Status 404 returned error can't find the container with id 575cba3d0c5cc232f7fb3aa2ce7a7130e08c26c9723e328133646d0c0fa61aeb Nov 02 15:22:43.383619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:43.383581 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" event={"ID":"07a33e49-9b35-47a8-a7f6-08d198062c1a","Type":"ContainerStarted","Data":"575cba3d0c5cc232f7fb3aa2ce7a7130e08c26c9723e328133646d0c0fa61aeb"} Nov 02 15:22:44.392736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.392702 5091 generic.go:356] "Generic (PLEG): container finished" podID="07a33e49-9b35-47a8-a7f6-08d198062c1a" containerID="4e4528d845313dae5ac5195026b7fc246ea41b243c100fdc6dc702bd6b4010b1" exitCode=0 Nov 02 15:22:44.393060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.392783 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" event={"ID":"07a33e49-9b35-47a8-a7f6-08d198062c1a","Type":"ContainerDied","Data":"4e4528d845313dae5ac5195026b7fc246ea41b243c100fdc6dc702bd6b4010b1"} Nov 02 15:22:44.511559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.505311 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b"] Nov 02 15:22:44.515942 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.515876 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9"] Nov 02 15:22:44.516739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.516711 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.522145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.522126 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.555308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.555278 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b"] Nov 02 15:22:44.605803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.605733 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9"] Nov 02 15:22:44.616945 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.616917 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:22:44.644152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644171 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644294 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644370 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrgl\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644459 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644496 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.644648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644592 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.644716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644696 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.644755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644743 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.644873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644857 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.644933 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.645006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.644992 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.645068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.645053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.645159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.645140 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.645205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.645175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smm28\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.645248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.645219 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.645283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.645268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.746823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.746791 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.746993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.746847 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.746993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.746912 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.746993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.746990 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.747140 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747125 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.747241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747217 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.747359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747244 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:44.747359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747314 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.747359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747351 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.747559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747379 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs\") pod \"17a5dc38-af8a-4235-9d70-ed61bb284a19\" (UID: \"17a5dc38-af8a-4235-9d70-ed61bb284a19\") " Nov 02 15:22:44.747647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:44.747711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:44.747769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747719 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.747821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747781 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.747821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747809 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.747927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747836 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.747927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747866 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrgl\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.747927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747871 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:44.748079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747938 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747979 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.747985 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:44.748079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748015 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748178 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748254 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748278 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748252 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748294 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748375 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748400 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748713 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.748877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748874 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smm28\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748941 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.748987 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748975 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.749145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.748986 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.749377 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.749656 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "kube-api-access-2qb8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.749709 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.749906 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.749998 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.750186 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.750222 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17a5dc38-af8a-4235-9d70-ed61bb284a19-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.750243 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.750244 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.750574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.750255 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.751138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.750967 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:44.751138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.751089 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:22:44.751489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.751462 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo" pod "17a5dc38-af8a-4235-9d70-ed61bb284a19" (UID: "17a5dc38-af8a-4235-9d70-ed61bb284a19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:22:44.752013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.751991 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.752119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.752026 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.752186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.752146 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.752512 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.752491 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.809629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.809594 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smm28\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.809807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.809776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.809934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.809909 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.810313 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.810293 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrgl\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl\") pod \"e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:44.847159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.847133 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:44.851902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.851873 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.851956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.851905 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.851956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.851914 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qb8q\" (UniqueName: \"kubernetes.io/projected/17a5dc38-af8a-4235-9d70-ed61bb284a19-kube-api-access-2qb8q\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.851956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.851924 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17a5dc38-af8a-4235-9d70-ed61bb284a19-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:44.855108 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:44.854896 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:45.290468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.290434 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b"] Nov 02 15:22:45.294266 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:45.294239 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d472de_908f_4f2f_8440_411c4b84f2c4.slice/crio-cddd82dba37ea70fcc456eb9fd4385479d893e6d8c3ea8ca6353daa7c819b6e3 WatchSource:0}: Error finding container cddd82dba37ea70fcc456eb9fd4385479d893e6d8c3ea8ca6353daa7c819b6e3: Status 404 returned error can't find the container with id cddd82dba37ea70fcc456eb9fd4385479d893e6d8c3ea8ca6353daa7c819b6e3 Nov 02 15:22:45.295850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.295829 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9"] Nov 02 15:22:45.297301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.297271 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.297386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.297328 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.297386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.297358 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.331939 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:45.331916 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2217a43f_8407_41ac_a256_4e55c7786769.slice/crio-8ab62c26de042e5f0b33e91f72f65d016c5dc04487061b2c4811e181daac6fb7 WatchSource:0}: Error finding container 8ab62c26de042e5f0b33e91f72f65d016c5dc04487061b2c4811e181daac6fb7: Status 404 returned error can't find the container with id 8ab62c26de042e5f0b33e91f72f65d016c5dc04487061b2c4811e181daac6fb7 Nov 02 15:22:45.334644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.334612 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.334755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.334676 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.334755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.334704 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:45.408166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.408128 5091 generic.go:356] "Generic (PLEG): container finished" podID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerID="d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca" exitCode=0 Nov 02 15:22:45.408729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.408274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" event={"ID":"17a5dc38-af8a-4235-9d70-ed61bb284a19","Type":"ContainerDied","Data":"d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca"} Nov 02 15:22:45.408729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.408325 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" event={"ID":"17a5dc38-af8a-4235-9d70-ed61bb284a19","Type":"ContainerDied","Data":"631860e3c378e860c7ece08bf6580289cf181f7e863376973fe1474c9be4284a"} Nov 02 15:22:45.408729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.408354 5091 scope.go:122] "RemoveContainer" containerID="d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca" Nov 02 15:22:45.408729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.408537 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9" Nov 02 15:22:45.411837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.411806 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" event={"ID":"2217a43f-8407-41ac-a256-4e55c7786769","Type":"ContainerStarted","Data":"07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb"} Nov 02 15:22:45.411933 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.411867 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" event={"ID":"2217a43f-8407-41ac-a256-4e55c7786769","Type":"ContainerStarted","Data":"8ab62c26de042e5f0b33e91f72f65d016c5dc04487061b2c4811e181daac6fb7"} Nov 02 15:22:45.415416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.415394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" event={"ID":"c5d472de-908f-4f2f-8440-411c4b84f2c4","Type":"ContainerStarted","Data":"b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a"} Nov 02 15:22:45.416723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.415718 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" event={"ID":"c5d472de-908f-4f2f-8440-411c4b84f2c4","Type":"ContainerStarted","Data":"cddd82dba37ea70fcc456eb9fd4385479d893e6d8c3ea8ca6353daa7c819b6e3"} Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.433390 5091 scope.go:122] "RemoveContainer" containerID="3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883" Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.446490 5091 scope.go:122] "RemoveContainer" containerID="d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca" Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:45.446894 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca\": container with ID starting with d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca not found: ID does not exist" containerID="d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca" Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.446932 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca"} err="failed to get container status \"d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca\": rpc error: code = NotFound desc = could not find container \"d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca\": container with ID starting with d11cc125eac4679fb912786533102a6e9ffad4eae55abc351cd56c8613a549ca not found: ID does not exist" Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.446958 5091 scope.go:122] "RemoveContainer" containerID="3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883" Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:45.447619 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883\": container with ID starting with 3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883 not found: ID does not exist" containerID="3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883" Nov 02 15:22:45.481096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.447647 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883"} err="failed to get container status \"3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883\": rpc error: code = NotFound desc = could not find container \"3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883\": container with ID starting with 3446cbd5aead362d6d84cf75259c4327c5ceb9726e0796514864bcc9a6021883 not found: ID does not exist" Nov 02 15:22:45.583369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.583182 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podStartSLOduration=1.583162669 podStartE2EDuration="1.583162669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:45.576798436 +0000 UTC m=+1519.741351001" watchObservedRunningTime="2026-11-02 15:22:45.583162669 +0000 UTC m=+1519.747715210" Nov 02 15:22:45.639584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.639274 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:45.694800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.694743 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podStartSLOduration=1.694728677 podStartE2EDuration="1.694728677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:45.692038287 +0000 UTC m=+1519.856590828" watchObservedRunningTime="2026-11-02 15:22:45.694728677 +0000 UTC m=+1519.859281218" Nov 02 15:22:45.773296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.769022 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca\") pod \"07a33e49-9b35-47a8-a7f6-08d198062c1a\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " Nov 02 15:22:45.773296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.769210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9\" (UniqueName: \"kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9\") pod \"07a33e49-9b35-47a8-a7f6-08d198062c1a\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " Nov 02 15:22:45.773296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.769316 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca\") pod \"07a33e49-9b35-47a8-a7f6-08d198062c1a\" (UID: \"07a33e49-9b35-47a8-a7f6-08d198062c1a\") " Nov 02 15:22:45.774325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.774295 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca" pod "07a33e49-9b35-47a8-a7f6-08d198062c1a" (UID: "07a33e49-9b35-47a8-a7f6-08d198062c1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:45.777266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.776611 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca" pod "07a33e49-9b35-47a8-a7f6-08d198062c1a" (UID: "07a33e49-9b35-47a8-a7f6-08d198062c1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:45.779052 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.779031 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9" pod "07a33e49-9b35-47a8-a7f6-08d198062c1a" (UID: "07a33e49-9b35-47a8-a7f6-08d198062c1a"). InnerVolumeSpecName "kube-api-access-v6kj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:45.803675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.803645 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9"] Nov 02 15:22:45.808890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.808868 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-k7pr9"] Nov 02 15:22:45.847497 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.847420 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:45.858315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.858289 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:45.871580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.871552 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:45.871580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.871578 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kj9\" (UniqueName: \"kubernetes.io/projected/07a33e49-9b35-47a8-a7f6-08d198062c1a-kube-api-access-v6kj9\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:45.871793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.871591 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07a33e49-9b35-47a8-a7f6-08d198062c1a-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:45.966964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:45.966925 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" path="/var/lib/kubelet/pods/17a5dc38-af8a-4235-9d70-ed61bb284a19/volumes" Nov 02 15:22:46.424427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.424399 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" Nov 02 15:22:46.424941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.424413 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk" event={"ID":"07a33e49-9b35-47a8-a7f6-08d198062c1a","Type":"ContainerDied","Data":"575cba3d0c5cc232f7fb3aa2ce7a7130e08c26c9723e328133646d0c0fa61aeb"} Nov 02 15:22:46.425350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.425053 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="575cba3d0c5cc232f7fb3aa2ce7a7130e08c26c9723e328133646d0c0fa61aeb" Nov 02 15:22:46.886724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.886686 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 15:22:46.887010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.886979 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 15:22:46.887378 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.886698 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 15:22:46.887559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:46.887506 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 15:22:46.923327 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:46.923296 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:22:47.850940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:47.850900 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 15:22:47.851555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:47.851500 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 15:22:47.861865 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:47.861837 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 15:22:47.862040 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:47.862016 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 15:22:48.369541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:48.359722 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:22:48.854595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:48.851903 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 15:22:48.854595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:48.851965 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 15:22:48.869557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:48.857844 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 15:22:48.869557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:48.857899 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 15:22:49.849901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:49.849864 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 15:22:49.850184 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:49.850155 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 15:22:49.861895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:49.861864 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 15:22:49.862432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:49.862404 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 15:22:50.850904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:50.850866 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 15:22:50.851155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:50.851127 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 15:22:50.858850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:50.858821 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 15:22:50.859037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:50.859016 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 15:22:51.848054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:51.848016 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 02 15:22:51.848580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:51.848066 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 02 15:22:51.858352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:51.858323 5091 patch_prober.go:29] interesting pod/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 02 15:22:51.858476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:51.858368 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 02 15:22:52.990591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:52.990559 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:52.992050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:52.992026 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:52.993211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:52.993196 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:22:53.111400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.111212 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:53.112991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.112643 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:53.114495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.114369 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:22:53.125570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.125547 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd"] Nov 02 15:22:53.128551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.126712 5091 cpu_manager.go:414] "containerMap: removing container" podUID="07a33e49-9b35-47a8-a7f6-08d198062c1a" containerName="revoke-keys" Nov 02 15:22:53.128698 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.128679 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a33e49-9b35-47a8-a7f6-08d198062c1a" containerName="revoke-keys" Nov 02 15:22:53.128880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.128864 5091 cpu_manager.go:414] "containerMap: removing container" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" Nov 02 15:22:53.128981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.128968 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" Nov 02 15:22:53.129084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.129073 5091 cpu_manager.go:414] "containerMap: removing container" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" Nov 02 15:22:53.129164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.129153 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" Nov 02 15:22:53.129484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.129465 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" Nov 02 15:22:53.129616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.129603 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a33e49-9b35-47a8-a7f6-08d198062c1a" containerName="revoke-keys" Nov 02 15:22:53.129724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.129713 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a5dc38-af8a-4235-9d70-ed61bb284a19" containerName="istio-proxy" Nov 02 15:22:53.135477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.135457 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.188906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.188872 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:53.189841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.189819 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:53.204640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.204612 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd"] Nov 02 15:22:53.292081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.291998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.292081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.292058 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.292285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.292260 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7fj\" (UniqueName: \"kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.393678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.393645 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.393841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.393726 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7fj\" (UniqueName: \"kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.393974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.393947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.394285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.394249 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.396587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.396567 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.477993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.477959 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7fj\" (UniqueName: \"kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj\") pod \"maas-api-revoke-sub-f8711a37-nh8zd\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:53.670455 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:53.670412 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache]" Nov 02 15:22:53.750909 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:53.750876 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:54.132792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:54.132759 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd"] Nov 02 15:22:54.136742 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:54.136707 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefabd006_4322_41f3_870a_5214e56d34d7.slice/crio-0f2f87a565d1a9f2835739871e4802c445e62bb0fed06110b49b50f7b6733510 WatchSource:0}: Error finding container 0f2f87a565d1a9f2835739871e4802c445e62bb0fed06110b49b50f7b6733510: Status 404 returned error can't find the container with id 0f2f87a565d1a9f2835739871e4802c445e62bb0fed06110b49b50f7b6733510 Nov 02 15:22:54.604900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:54.604868 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" event={"ID":"efabd006-4322-41f3-870a-5214e56d34d7","Type":"ContainerStarted","Data":"ffe2743bdd2ba06e95eb406e17b2313de9fac0d16e166e97bab9077b08c4ce4a"} Nov 02 15:22:54.605039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:54.604932 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" event={"ID":"efabd006-4322-41f3-870a-5214e56d34d7","Type":"ContainerStarted","Data":"0f2f87a565d1a9f2835739871e4802c445e62bb0fed06110b49b50f7b6733510"} Nov 02 15:22:54.719056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:54.719007 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" podStartSLOduration=1.71899444 podStartE2EDuration="1.71899444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:54.717804605 +0000 UTC m=+1528.882357144" watchObservedRunningTime="2026-11-02 15:22:54.71899444 +0000 UTC m=+1528.883546977" Nov 02 15:22:55.613817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:55.613734 5091 generic.go:356] "Generic (PLEG): container finished" podID="efabd006-4322-41f3-870a-5214e56d34d7" containerID="ffe2743bdd2ba06e95eb406e17b2313de9fac0d16e166e97bab9077b08c4ce4a" exitCode=0 Nov 02 15:22:55.613817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:55.613792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" event={"ID":"efabd006-4322-41f3-870a-5214e56d34d7","Type":"ContainerDied","Data":"ffe2743bdd2ba06e95eb406e17b2313de9fac0d16e166e97bab9077b08c4ce4a"} Nov 02 15:22:56.387510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.387472 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b"] Nov 02 15:22:56.398470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.393140 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.445811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.445756 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-89017f75-openshift-default-dockercfg-6r78f\"" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.558709 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.558811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.558879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.558913 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.558996 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.559071 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.559108 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-599bj\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.559187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.581540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.559222 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.636125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.636039 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b"] Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.698904 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.698992 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699028 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699119 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699234 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-599bj\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699303 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699331 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.699477 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.700673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.700409 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.701723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.701700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.702238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.702220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.704635 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.704606 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.704876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.704855 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.751076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.749601 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.767202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.767167 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.790542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.783667 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-599bj\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.790542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.787289 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token\") pod \"e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.818280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:56.816898 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:56.979715 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:56.978947 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache]" Nov 02 15:22:57.085467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.085051 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:57.129098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.129065 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca\") pod \"efabd006-4322-41f3-870a-5214e56d34d7\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " Nov 02 15:22:57.129254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.129119 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj\" (UniqueName: \"kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj\") pod \"efabd006-4322-41f3-870a-5214e56d34d7\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " Nov 02 15:22:57.129254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.129180 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca\") pod \"efabd006-4322-41f3-870a-5214e56d34d7\" (UID: \"efabd006-4322-41f3-870a-5214e56d34d7\") " Nov 02 15:22:57.129893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.129865 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca" pod "efabd006-4322-41f3-870a-5214e56d34d7" (UID: "efabd006-4322-41f3-870a-5214e56d34d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:22:57.131800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.131778 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj" pod "efabd006-4322-41f3-870a-5214e56d34d7" (UID: "efabd006-4322-41f3-870a-5214e56d34d7"). InnerVolumeSpecName "kube-api-access-qp7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:22:57.132072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.132047 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca" pod "efabd006-4322-41f3-870a-5214e56d34d7" (UID: "efabd006-4322-41f3-870a-5214e56d34d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:22:57.241419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.240430 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.241419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.240466 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7fj\" (UniqueName: \"kubernetes.io/projected/efabd006-4322-41f3-870a-5214e56d34d7-kube-api-access-qp7fj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.241419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.240485 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efabd006-4322-41f3-870a-5214e56d34d7-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:22:57.370980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.370951 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b"] Nov 02 15:22:57.374012 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:22:57.373981 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c381744_eba9_4e6b_ae0a_c13e68c631ac.slice/crio-56b96b0e54f97c11e9c13d678af5540232f5080e30e91f0dd70ed453a5ec02b6 WatchSource:0}: Error finding container 56b96b0e54f97c11e9c13d678af5540232f5080e30e91f0dd70ed453a5ec02b6: Status 404 returned error can't find the container with id 56b96b0e54f97c11e9c13d678af5540232f5080e30e91f0dd70ed453a5ec02b6 Nov 02 15:22:57.377124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.377088 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:57.377235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.377163 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:57.377235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.377210 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:22:57.775491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.775459 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" Nov 02 15:22:57.775967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.775662 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd" event={"ID":"efabd006-4322-41f3-870a-5214e56d34d7","Type":"ContainerDied","Data":"0f2f87a565d1a9f2835739871e4802c445e62bb0fed06110b49b50f7b6733510"} Nov 02 15:22:57.775967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.775705 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f2f87a565d1a9f2835739871e4802c445e62bb0fed06110b49b50f7b6733510" Nov 02 15:22:57.778243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.778211 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" event={"ID":"4c381744-eba9-4e6b-ae0a-c13e68c631ac","Type":"ContainerStarted","Data":"27efc52205b8263c43672f6c4737caa5f1e24cf596e9c2fbcdbbab21c0070d1a"} Nov 02 15:22:57.778393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.778264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" event={"ID":"4c381744-eba9-4e6b-ae0a-c13e68c631ac","Type":"ContainerStarted","Data":"56b96b0e54f97c11e9c13d678af5540232f5080e30e91f0dd70ed453a5ec02b6"} Nov 02 15:22:57.818361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.818332 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:57.896659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:57.896604 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podStartSLOduration=1.896583393 podStartE2EDuration="1.896583393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:22:57.895006749 +0000 UTC m=+1532.059559292" watchObservedRunningTime="2026-11-02 15:22:57.896583393 +0000 UTC m=+1532.061135936" Nov 02 15:22:57.943602 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:22:57.943462 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:22:58.824714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:58.824680 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:59.747796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.747747 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl"] Nov 02 15:22:59.748829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.748804 5091 cpu_manager.go:414] "containerMap: removing container" podUID="efabd006-4322-41f3-870a-5214e56d34d7" containerName="revoke-keys" Nov 02 15:22:59.748829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.748829 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efabd006-4322-41f3-870a-5214e56d34d7" containerName="revoke-keys" Nov 02 15:22:59.749044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.749032 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="efabd006-4322-41f3-870a-5214e56d34d7" containerName="revoke-keys" Nov 02 15:22:59.753580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.753561 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:22:59.817536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.811695 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:22:59.817536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.811779 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:22:59.817536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.812584 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl"] Nov 02 15:22:59.820991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.820960 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:59.822172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.822148 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:22:59.892738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.892699 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54nhb\" (UniqueName: \"kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:22:59.893253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.892925 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:22:59.893253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.892970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:22:59.997539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.994828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:22:59.997539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.994896 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:22:59.997539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:22:59.996418 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:00.011815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:00.011092 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54nhb\" (UniqueName: \"kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:00.076676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:00.076585 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:00.097385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:00.090273 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54nhb\" (UniqueName: \"kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb\") pod \"maas-api-revoke-sub-a6eb75e8-f75gl\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:00.373734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:00.373418 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:00.908654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:00.904073 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl"] Nov 02 15:23:01.833429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:01.833394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" event={"ID":"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b","Type":"ContainerStarted","Data":"615bb9e7ec4dd3bed701ddfe8e3857e3f120adf4694c1c5a6116b07479d897e8"} Nov 02 15:23:01.833649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:01.833454 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" event={"ID":"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b","Type":"ContainerStarted","Data":"5f26254758ec99530ce2182db721c83d0153003e131834dd525ff130295bcfee"} Nov 02 15:23:01.949006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:01.948958 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" podStartSLOduration=2.948944182 podStartE2EDuration="2.948944182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:01.945684175 +0000 UTC m=+1536.110236720" watchObservedRunningTime="2026-11-02 15:23:01.948944182 +0000 UTC m=+1536.113496722" Nov 02 15:23:02.851663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:02.851621 5091 generic.go:356] "Generic (PLEG): container finished" podID="c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" containerID="615bb9e7ec4dd3bed701ddfe8e3857e3f120adf4694c1c5a6116b07479d897e8" exitCode=0 Nov 02 15:23:02.851880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:02.851779 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" event={"ID":"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b","Type":"ContainerDied","Data":"615bb9e7ec4dd3bed701ddfe8e3857e3f120adf4694c1c5a6116b07479d897e8"} Nov 02 15:23:04.186884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.186852 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:04.290562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.288322 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca\") pod \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " Nov 02 15:23:04.290562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.288360 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca\") pod \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " Nov 02 15:23:04.290562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.288466 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb\" (UniqueName: \"kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb\") pod \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\" (UID: \"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b\") " Nov 02 15:23:04.290562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.289189 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca" pod "c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" (UID: "c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:04.290562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.289860 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:23:04.319782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.317036 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb" pod "c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" (UID: "c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b"). InnerVolumeSpecName "kube-api-access-54nhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:04.319782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.317531 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca" pod "c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" (UID: "c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:23:04.334597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.325004 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn"] Nov 02 15:23:04.334597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.325904 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" containerName="revoke-keys" Nov 02 15:23:04.334597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.325922 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" containerName="revoke-keys" Nov 02 15:23:04.334597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.326117 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" containerName="revoke-keys" Nov 02 15:23:04.334597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.330358 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.383329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.380549 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-89017f75\"" Nov 02 15:23:04.383329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.380711 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-89017f75-dockercfg-wt64f\"" Nov 02 15:23:04.388893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.388723 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn"] Nov 02 15:23:04.392337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.392263 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqfv\" (UniqueName: \"kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv\") pod \"payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.399688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.395023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.399688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.395566 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54nhb\" (UniqueName: \"kubernetes.io/projected/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-kube-api-access-54nhb\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:23:04.399688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.395590 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:23:04.499504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.498054 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqfv\" (UniqueName: \"kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv\") pod \"payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.499504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.498127 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.499504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.499041 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume\") pod \"payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.560289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.560256 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqfv\" (UniqueName: \"kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv\") pod \"payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.670923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.670759 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:04.873301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.873271 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" Nov 02 15:23:04.873483 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.873309 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl" event={"ID":"c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b","Type":"ContainerDied","Data":"5f26254758ec99530ce2182db721c83d0153003e131834dd525ff130295bcfee"} Nov 02 15:23:04.873483 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:04.873355 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f26254758ec99530ce2182db721c83d0153003e131834dd525ff130295bcfee" Nov 02 15:23:05.106129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:05.106053 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn"] Nov 02 15:23:05.109817 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:23:05.109792 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c7aaf7_0dee_4287_9686_1d5d26ad93d5.slice/crio-bd3966b9cac99a4c28968c77eafe1126c3f59ec766d1728a940b6a7cd12b9170 WatchSource:0}: Error finding container bd3966b9cac99a4c28968c77eafe1126c3f59ec766d1728a940b6a7cd12b9170: Status 404 returned error can't find the container with id bd3966b9cac99a4c28968c77eafe1126c3f59ec766d1728a940b6a7cd12b9170 Nov 02 15:23:05.883429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:05.883380 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" event={"ID":"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5","Type":"ContainerStarted","Data":"b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470"} Nov 02 15:23:05.883868 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:05.883457 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" event={"ID":"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5","Type":"ContainerStarted","Data":"bd3966b9cac99a4c28968c77eafe1126c3f59ec766d1728a940b6a7cd12b9170"} Nov 02 15:23:05.883868 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:05.883542 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:06.000774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:06.000716 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" podStartSLOduration=2.000702565 podStartE2EDuration="2.000702565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:05.99620425 +0000 UTC m=+1540.160756789" watchObservedRunningTime="2026-11-02 15:23:06.000702565 +0000 UTC m=+1540.165255116" Nov 02 15:23:07.089795 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:07.089749 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:23:08.609265 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:08.609219 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:23:10.623526 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.623480 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf"] Nov 02 15:23:10.629012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.628991 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.683627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.683597 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf"] Nov 02 15:23:10.724383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.724357 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:23:10.724562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.724483 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:23:10.795488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.795453 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.795658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.795605 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.795750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.795737 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hbf\" (UniqueName: \"kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.897542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.897443 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.897696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.897610 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hbf\" (UniqueName: \"kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.897798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.897699 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.898277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.898249 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.900203 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.900179 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:10.955917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:10.955891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hbf\" (UniqueName: \"kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf\") pod \"maas-api-revoke-sub-f165ee86-h9sgf\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:11.244092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:11.244017 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:11.624133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:11.624111 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf"] Nov 02 15:23:11.627036 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:23:11.627005 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6512fd_d798_4863_ae51_2db0e27fad70.slice/crio-6ecb19550c0a71779f511b8a05f2ced8f0140f412ec7767df4e1c02a4c0feeb2 WatchSource:0}: Error finding container 6ecb19550c0a71779f511b8a05f2ced8f0140f412ec7767df4e1c02a4c0feeb2: Status 404 returned error can't find the container with id 6ecb19550c0a71779f511b8a05f2ced8f0140f412ec7767df4e1c02a4c0feeb2 Nov 02 15:23:11.933075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:11.933045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" event={"ID":"3b6512fd-d798-4863-ae51-2db0e27fad70","Type":"ContainerStarted","Data":"6ecb19550c0a71779f511b8a05f2ced8f0140f412ec7767df4e1c02a4c0feeb2"} Nov 02 15:23:12.944220 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:12.944184 5091 generic.go:356] "Generic (PLEG): container finished" podID="3b6512fd-d798-4863-ae51-2db0e27fad70" containerID="6ede590e0e69f10be3d40451c7e2610686cd1d4a70377b53a325700aa089a217" exitCode=0 Nov 02 15:23:12.944656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:12.944278 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" event={"ID":"3b6512fd-d798-4863-ae51-2db0e27fad70","Type":"ContainerDied","Data":"6ede590e0e69f10be3d40451c7e2610686cd1d4a70377b53a325700aa089a217"} Nov 02 15:23:14.140199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.140172 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:14.242835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.242747 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca\") pod \"3b6512fd-d798-4863-ae51-2db0e27fad70\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " Nov 02 15:23:14.242835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.242825 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca\") pod \"3b6512fd-d798-4863-ae51-2db0e27fad70\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " Nov 02 15:23:14.243075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.242909 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf\" (UniqueName: \"kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf\") pod \"3b6512fd-d798-4863-ae51-2db0e27fad70\" (UID: \"3b6512fd-d798-4863-ae51-2db0e27fad70\") " Nov 02 15:23:14.243281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.243248 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca" pod "3b6512fd-d798-4863-ae51-2db0e27fad70" (UID: "3b6512fd-d798-4863-ae51-2db0e27fad70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:23:14.243760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.243742 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:23:14.245694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.245667 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf" pod "3b6512fd-d798-4863-ae51-2db0e27fad70" (UID: "3b6512fd-d798-4863-ae51-2db0e27fad70"). InnerVolumeSpecName "kube-api-access-96hbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:23:14.248871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.248847 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca" pod "3b6512fd-d798-4863-ae51-2db0e27fad70" (UID: "3b6512fd-d798-4863-ae51-2db0e27fad70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:23:14.345632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.345595 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6512fd-d798-4863-ae51-2db0e27fad70-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:23:14.345632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.345633 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hbf\" (UniqueName: \"kubernetes.io/projected/3b6512fd-d798-4863-ae51-2db0e27fad70-kube-api-access-96hbf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:23:14.962270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.962238 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" Nov 02 15:23:14.962455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.962285 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf" event={"ID":"3b6512fd-d798-4863-ae51-2db0e27fad70","Type":"ContainerDied","Data":"6ecb19550c0a71779f511b8a05f2ced8f0140f412ec7767df4e1c02a4c0feeb2"} Nov 02 15:23:14.962455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:14.962345 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ecb19550c0a71779f511b8a05f2ced8f0140f412ec7767df4e1c02a4c0feeb2" Nov 02 15:23:16.895157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:16.895127 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:23:16.913464 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:16.913431 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache]" Nov 02 15:23:16.913632 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:16.913438 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache]" Nov 02 15:23:17.106947 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:17.106912 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:23:23.684971 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:23.684934 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e938984_35f5_4011_a10b_b0c8ae2d72b7.slice/crio-f9420ea36d28bc4ff0500f67257360dc5ca857a8e83f39070ba7dee2e3eb554c\": RecentStats: unable to find data in memory cache]" Nov 02 15:23:27.938349 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:27.938303 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:23:28.559566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:28.559534 5091 scope.go:122] "RemoveContainer" containerID="a8767940fdc18c7dfb77340a69164ecf1e5861022efaba9af06695a72a2c2a6b" Nov 02 15:23:28.572393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:28.572369 5091 scope.go:122] "RemoveContainer" containerID="49983950a47ba0dd722649a1dfb2134460cbf4dd473ec921de628a5affb6d57c" Nov 02 15:23:28.582334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:28.582313 5091 scope.go:122] "RemoveContainer" containerID="57fb3230445f3aeff806844e1756f7c4c1f9056323d8786fe02ffe024b3a107e" Nov 02 15:23:28.591333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:28.591310 5091 scope.go:122] "RemoveContainer" containerID="c0c62f63b89a5d34f41e9f5bb7cc3c3f8ca7f41895ecd72693ab49f59ab0a479" Nov 02 15:23:29.919189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:29.919153 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs"] Nov 02 15:23:29.919825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:29.919811 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3b6512fd-d798-4863-ae51-2db0e27fad70" containerName="revoke-keys" Nov 02 15:23:29.919873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:29.919826 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6512fd-d798-4863-ae51-2db0e27fad70" containerName="revoke-keys" Nov 02 15:23:29.919967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:29.919958 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6512fd-d798-4863-ae51-2db0e27fad70" containerName="revoke-keys" Nov 02 15:23:29.925186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:29.925165 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:29.980796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:29.980765 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs"] Nov 02 15:23:30.021339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.021303 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-06a527d4-serving-cert\"" Nov 02 15:23:30.105670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.105633 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-497mp\" (UniqueName: \"kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.105879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.105855 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.106052 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.106031 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.106161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.106081 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.211147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.211039 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.211923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.211894 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.212646 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.212624 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-497mp\" (UniqueName: \"kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.213209 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.213188 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.214640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.214102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.221276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.221250 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.223772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.223746 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.288548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.285854 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-497mp\" (UniqueName: \"kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp\") pod \"maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:30.543950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:30.543860 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:31.702150 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:31.702115 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs"] Nov 02 15:23:32.417546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.414603 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 02 15:23:32.421540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.416912 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Nov 02 15:23:32.421540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.418647 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.486725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.486600 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.487343 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.487307 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 02 15:23:32.492544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.487820 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:23:32.520540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.493019 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:23:32.520845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.512951 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.693274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.693217 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.694011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.693552 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 02 15:23:32.782425 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.782329 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:32.784192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.784162 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" event={"ID":"8646943b-1e40-4354-9d88-716d1abe31fd","Type":"ContainerStarted","Data":"26154d9743d0eda5cf3ea1963e49b160bbb60d9ba2a716b05fb887b2b552933a"} Nov 02 15:23:32.808655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:32.808609 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:33.061712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:33.061616 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:23:33.796081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:33.795997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" event={"ID":"8646943b-1e40-4354-9d88-716d1abe31fd","Type":"ContainerStarted","Data":"eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d"} Nov 02 15:23:33.796505 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:33.796101 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:33.912937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:33.912871 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" podStartSLOduration=3.380866086 podStartE2EDuration="4.912852863s" totalImagesPullingTime="1.531986777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:33.911543806 +0000 UTC m=+1568.076096348" watchObservedRunningTime="2026-11-02 15:23:33.912852863 +0000 UTC m=+1568.077405405" Nov 02 15:23:39.811934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:39.811844 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:23:54.896368 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.896332 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7"] Nov 02 15:23:54.900754 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.900719 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922396 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922448 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922637 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98t4m\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922687 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922780 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922825 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922909 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922962 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922951 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.922997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.922974 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:54.951076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:54.951050 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x6s8k\"" Nov 02 15:23:55.004121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.004085 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7"] Nov 02 15:23:55.024236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.024170 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.024412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.024247 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.024685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.024660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.026357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026332 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98t4m\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.026484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026403 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.026484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026471 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.026681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026557 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.026681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026643 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.026864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026683 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.027123 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.026714 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.027500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.027474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.027618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.027578 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.028101 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.028074 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.028101 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.028077 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.028691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.028664 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.031551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.030556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.086624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.086590 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.086863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.086841 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98t4m\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m\") pod \"maas-default-gateway-openshift-default-7f5667f86c-bf9z7\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.214838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.214753 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:55.655276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.655241 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7"] Nov 02 15:23:55.657339 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:23:55.657314 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b85cbdb_53a9_4259_9c5c_b1c14e672e5d.slice/crio-0aceb0d2b98615bdd5f13a6c1467b648d7b51cc8ec69f04a197a09a7ab7f4722 WatchSource:0}: Error finding container 0aceb0d2b98615bdd5f13a6c1467b648d7b51cc8ec69f04a197a09a7ab7f4722: Status 404 returned error can't find the container with id 0aceb0d2b98615bdd5f13a6c1467b648d7b51cc8ec69f04a197a09a7ab7f4722 Nov 02 15:23:55.660172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.660139 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:23:55.660282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.660214 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:23:55.660282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.660260 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:23:55.998258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.998178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" event={"ID":"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d","Type":"ContainerStarted","Data":"d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07"} Nov 02 15:23:55.998258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:55.998230 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" event={"ID":"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d","Type":"ContainerStarted","Data":"0aceb0d2b98615bdd5f13a6c1467b648d7b51cc8ec69f04a197a09a7ab7f4722"} Nov 02 15:23:56.117684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:56.117630 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podStartSLOduration=2.117616222 podStartE2EDuration="2.117616222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:23:56.113469742 +0000 UTC m=+1590.278022308" watchObservedRunningTime="2026-11-02 15:23:56.117616222 +0000 UTC m=+1590.282168762" Nov 02 15:23:56.215427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:56.215388 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:23:56.216879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:56.216852 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:23:56.217005 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:56.216900 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:23:57.215488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:57.215445 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:23:57.215898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:57.215510 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:23:57.967872 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:23:57.967835 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:23:58.215374 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:58.215336 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:23:58.215568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:58.215389 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:23:59.215380 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:59.215343 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:23:59.215584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:23:59.215398 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:24:00.219542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:00.218957 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:24:00.219542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:00.219034 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:24:01.215611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:01.215570 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:24:01.215829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:01.215638 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:24:02.235145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:02.218824 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:24:02.235145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:02.218894 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:24:02.493599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:02.487033 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 02 15:24:02.638906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:02.638267 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.068569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.060739 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.231669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.231630 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:24:03.280666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.280616 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:24:03.434542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.410602 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.434542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.433991 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.591542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.525009 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:24:03.591542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.525315 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": context deadline exceeded" Nov 02 15:24:03.591542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.534797 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.617294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.617181 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.642690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.642615 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:03.644418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.644394 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:24:03.719284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:03.718642 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:04.218882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.218844 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 02 15:24:04.219194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.219161 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 02 15:24:04.438538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.434612 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:04.438538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.435229 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:24:04.540315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.538646 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:04.540315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.539255 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:24:04.691991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:04.688633 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:05.437667 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.437620 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:05.464656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.464608 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:05.485649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.485600 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:05.486272 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.486236 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 02 15:24:05.542722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.542665 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:05.664973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.664931 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:24:05.686553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:05.685350 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:06.146567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:06.131913 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:24:06.265548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:06.263700 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:24:06.265548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:06.263779 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:06.308116 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:06.306786 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:24:07.275594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:07.265616 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:24:07.275594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:07.265691 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:07.595187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:07.595087 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:24:07.601850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:07.597191 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:24:07.603134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:07.603099 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:24:10.045166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.045125 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85"] Nov 02 15:24:10.052077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.052052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.163603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.163566 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85"] Nov 02 15:24:10.198024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.197891 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.197992 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198019 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198072 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee194769-7328-456a-b24d-c368212d5672-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198134 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198166 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee194769-7328-456a-b24d-c368212d5672-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198196 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bjtw\" (UniqueName: \"kubernetes.io/projected/ee194769-7328-456a-b24d-c368212d5672-kube-api-access-2bjtw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.198324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.198239 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee194769-7328-456a-b24d-c368212d5672-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300173 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee194769-7328-456a-b24d-c368212d5672-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300296 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300344 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee194769-7328-456a-b24d-c368212d5672-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300388 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bjtw\" (UniqueName: \"kubernetes.io/projected/ee194769-7328-456a-b24d-c368212d5672-kube-api-access-2bjtw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300782 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee194769-7328-456a-b24d-c368212d5672-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.300986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.300942 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.301249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.301075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.301249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.301117 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.301363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.301342 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee194769-7328-456a-b24d-c368212d5672-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.301591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.301566 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.301677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.301641 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.301911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.301892 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.303038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.303015 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee194769-7328-456a-b24d-c368212d5672-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.304262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.304240 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee194769-7328-456a-b24d-c368212d5672-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.360370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.360281 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee194769-7328-456a-b24d-c368212d5672-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.360741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.360712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bjtw\" (UniqueName: \"kubernetes.io/projected/ee194769-7328-456a-b24d-c368212d5672-kube-api-access-2bjtw\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r2p85\" (UID: \"ee194769-7328-456a-b24d-c368212d5672\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.369460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.369415 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:10.826311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.826275 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85"] Nov 02 15:24:10.835426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.835388 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:10.835571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.835481 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:10.835571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:10.835542 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:11.809051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:11.807845 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" event={"ID":"ee194769-7328-456a-b24d-c368212d5672","Type":"ContainerStarted","Data":"669ff8f7b4f6c2057d2346c64aa0c23aaaa05e240227ee2603388ed488fef872"} Nov 02 15:24:11.809051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:11.807909 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" event={"ID":"ee194769-7328-456a-b24d-c368212d5672","Type":"ContainerStarted","Data":"9fed62191767599baa4cef3997b14791ac312f91a4949c80f63e59289ed813f7"} Nov 02 15:24:12.063594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:12.053646 5091 patch_prober.go:29] interesting pod/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:24:12.063594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:12.053711 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:24:12.374063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:12.373986 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:12.376212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:12.375821 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:12.376212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:12.375902 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:13.369811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.369771 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:13.370330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.369832 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:13.825931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.825904 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerID="2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18" exitCode=0 Nov 02 15:24:13.844774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.825972 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" event={"ID":"c1f21248-035b-4e39-9691-73f0b5a4e373","Type":"ContainerDied","Data":"2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18"} Nov 02 15:24:13.844774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.826372 5091 scope.go:122] "RemoveContainer" containerID="2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18" Nov 02 15:24:13.847976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.846631 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:13.847976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.846727 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:13.847976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.846776 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:13.989419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:13.989381 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podStartSLOduration=4.989368355 podStartE2EDuration="4.989368355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:11.980814051 +0000 UTC m=+1606.145366592" watchObservedRunningTime="2026-11-02 15:24:13.989368355 +0000 UTC m=+1608.153920895" Nov 02 15:24:14.370710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.370672 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:14.371238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.370736 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:14.886236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.886193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" event={"ID":"c1f21248-035b-4e39-9691-73f0b5a4e373","Type":"ContainerStarted","Data":"c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a"} Nov 02 15:24:14.890545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.890057 5091 generic.go:356] "Generic (PLEG): container finished" podID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerID="b78050d8245551571d6eda65ea53e3386ec49d1954cb8e2194fe9598c12bcf08" exitCode=0 Nov 02 15:24:14.890545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.890215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" event={"ID":"166421cc-2839-41e7-8ac8-223db1bdc0de","Type":"ContainerDied","Data":"b78050d8245551571d6eda65ea53e3386ec49d1954cb8e2194fe9598c12bcf08"} Nov 02 15:24:14.890764 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.890692 5091 scope.go:122] "RemoveContainer" containerID="b78050d8245551571d6eda65ea53e3386ec49d1954cb8e2194fe9598c12bcf08" Nov 02 15:24:14.893823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.893788 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:14.894078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.894059 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:14.894226 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:14.894208 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:15.302058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.301921 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:24:15.395535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.394969 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:15.395535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.395040 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:15.867373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.866933 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:24:15.867373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.867066 5091 patch_prober.go:29] interesting pod/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:24:15.867373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.867112 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 15:24:15.923668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:15.923624 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" event={"ID":"166421cc-2839-41e7-8ac8-223db1bdc0de","Type":"ContainerStarted","Data":"416b3e51a1b45a2423c2ef2f58597da61b4dc22c016f025c2edbb7d64e8b9d1c"} Nov 02 15:24:16.261599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.260894 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz"] Nov 02 15:24:16.277541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.276048 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.323268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.323177 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz"] Nov 02 15:24:16.342538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.340443 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:24:16.342538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.340599 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:24:16.374538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.370899 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:16.374538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.370964 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:16.450094 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.450049 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c7xj\" (UniqueName: \"kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.457661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.450245 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.457661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.450283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.533606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.533384 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:24:16.551551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.550654 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:24:16.551551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.550719 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 15:24:16.606540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.553167 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c7xj\" (UniqueName: \"kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.606540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.606365 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.606540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.606431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.623623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.612854 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.623623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.613561 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.741547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.734603 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c7xj\" (UniqueName: \"kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj\") pod \"maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:16.886542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.872712 5091 patch_prober.go:29] interesting pod/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:24:16.886542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.872815 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 15:24:16.925417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:16.925385 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:17.393704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.392999 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:17.393704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.393074 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:17.521384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.521345 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:24:17.521862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.521410 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 15:24:17.697786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.694657 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz"] Nov 02 15:24:17.868308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.867812 5091 patch_prober.go:29] interesting pod/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:24:17.868308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:17.867874 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 15:24:18.009715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.009511 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" event={"ID":"17381802-8e4f-4754-a2e1-69c517427016","Type":"ContainerStarted","Data":"ca88d67a15c71a2dbb3cf67803152e54fed955f28ab6a64f102ef20800749847"} Nov 02 15:24:18.385541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.372415 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:18.385541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.372477 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:18.521455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.521363 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:24:18.522066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.522032 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 15:24:18.878601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.864881 5091 patch_prober.go:29] interesting pod/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:24:18.878601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:18.864937 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 15:24:19.018499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.018460 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" event={"ID":"17381802-8e4f-4754-a2e1-69c517427016","Type":"ContainerStarted","Data":"41b2df95b6a7604fe6ccd7d34499e81532fcbe387767ffc8ea8431dfa136a862"} Nov 02 15:24:19.181598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.174782 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" podStartSLOduration=3.1747599109999998 podStartE2EDuration="3.174759911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:19.172497353 +0000 UTC m=+1613.337049894" watchObservedRunningTime="2026-11-02 15:24:19.174759911 +0000 UTC m=+1613.339312454" Nov 02 15:24:19.371540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.371044 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body= Nov 02 15:24:19.371540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.371128 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" Nov 02 15:24:19.521684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.521602 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:24:19.522958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.522430 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 15:24:19.868539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.865922 5091 patch_prober.go:29] interesting pod/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 02 15:24:19.868539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:19.865987 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 02 15:24:20.118598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.116647 5091 generic.go:356] "Generic (PLEG): container finished" podID="17381802-8e4f-4754-a2e1-69c517427016" containerID="41b2df95b6a7604fe6ccd7d34499e81532fcbe387767ffc8ea8431dfa136a862" exitCode=0 Nov 02 15:24:20.120163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.120125 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" event={"ID":"17381802-8e4f-4754-a2e1-69c517427016","Type":"ContainerDied","Data":"41b2df95b6a7604fe6ccd7d34499e81532fcbe387767ffc8ea8431dfa136a862"} Nov 02 15:24:20.521712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.521478 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:24:20.521712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.521574 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 15:24:20.878462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.878430 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:24:20.879364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.879334 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:24:20.880331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.880308 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:24:20.992238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.992170 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:20.993043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.992854 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:20.993643 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:20.993621 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" Nov 02 15:24:21.488116 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.488086 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:21.521824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.521785 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Nov 02 15:24:21.522215 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.521838 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Nov 02 15:24:21.590234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.590200 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca\") pod \"17381802-8e4f-4754-a2e1-69c517427016\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " Nov 02 15:24:21.590431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.590331 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj\" (UniqueName: \"kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj\") pod \"17381802-8e4f-4754-a2e1-69c517427016\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " Nov 02 15:24:21.590431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.590423 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca\") pod \"17381802-8e4f-4754-a2e1-69c517427016\" (UID: \"17381802-8e4f-4754-a2e1-69c517427016\") " Nov 02 15:24:21.590812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.590784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca" pod "17381802-8e4f-4754-a2e1-69c517427016" (UID: "17381802-8e4f-4754-a2e1-69c517427016"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:21.591081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.591059 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17381802-8e4f-4754-a2e1-69c517427016-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:21.592484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.592460 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj" pod "17381802-8e4f-4754-a2e1-69c517427016" (UID: "17381802-8e4f-4754-a2e1-69c517427016"). InnerVolumeSpecName "kube-api-access-2c7xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:21.592580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.592490 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca" pod "17381802-8e4f-4754-a2e1-69c517427016" (UID: "17381802-8e4f-4754-a2e1-69c517427016"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:24:21.692887 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.692854 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17381802-8e4f-4754-a2e1-69c517427016-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:21.692887 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:21.692885 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c7xj\" (UniqueName: \"kubernetes.io/projected/17381802-8e4f-4754-a2e1-69c517427016-kube-api-access-2c7xj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:22.207148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:22.207055 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" event={"ID":"17381802-8e4f-4754-a2e1-69c517427016","Type":"ContainerDied","Data":"ca88d67a15c71a2dbb3cf67803152e54fed955f28ab6a64f102ef20800749847"} Nov 02 15:24:22.207148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:22.207112 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz" Nov 02 15:24:22.207148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:22.207120 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca88d67a15c71a2dbb3cf67803152e54fed955f28ab6a64f102ef20800749847" Nov 02 15:24:22.526362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:22.526333 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:24:22.526761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:22.526746 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:24:22.527354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:22.527337 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:24:27.938145 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:24:27.938102 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:24:27.994099 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:27.994061 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522"] Nov 02 15:24:29.346505 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.346470 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz"] Nov 02 15:24:29.347214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.347197 5091 cpu_manager.go:414] "containerMap: removing container" podUID="17381802-8e4f-4754-a2e1-69c517427016" containerName="revoke-keys" Nov 02 15:24:29.347263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.347214 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17381802-8e4f-4754-a2e1-69c517427016" containerName="revoke-keys" Nov 02 15:24:29.347382 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.347373 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="17381802-8e4f-4754-a2e1-69c517427016" containerName="revoke-keys" Nov 02 15:24:29.352132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.352115 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.401978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.401938 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:24:29.412794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.412764 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz"] Nov 02 15:24:29.492543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.455682 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:24:29.492543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.457140 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.492543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.457205 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56g4q\" (UniqueName: \"kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.492543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.457317 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.562536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.562425 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.562536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.562492 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56g4q\" (UniqueName: \"kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.562803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.562632 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.565676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.563800 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.599886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.599816 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.652482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.652438 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56g4q\" (UniqueName: \"kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q\") pod \"maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:29.666855 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:29.666823 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:30.056896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:30.056320 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz"] Nov 02 15:24:30.059469 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:24:30.059437 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5b0265_500b_45bf_9fad_e7ffcf0fdf3c.slice/crio-f1a9f448e53f03fd034e89435a42b2f04fa11815219b64545d0e0ac4e0028745 WatchSource:0}: Error finding container f1a9f448e53f03fd034e89435a42b2f04fa11815219b64545d0e0ac4e0028745: Status 404 returned error can't find the container with id f1a9f448e53f03fd034e89435a42b2f04fa11815219b64545d0e0ac4e0028745 Nov 02 15:24:30.282614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:30.282559 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" event={"ID":"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c","Type":"ContainerStarted","Data":"f1a9f448e53f03fd034e89435a42b2f04fa11815219b64545d0e0ac4e0028745"} Nov 02 15:24:31.291584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:31.291494 5091 generic.go:356] "Generic (PLEG): container finished" podID="0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" containerID="9d72811a9b7270d42e7b9b833d497468d2da008e26c52db3b119b06e90ef8d03" exitCode=0 Nov 02 15:24:31.291962 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:31.291573 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" event={"ID":"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c","Type":"ContainerDied","Data":"9d72811a9b7270d42e7b9b833d497468d2da008e26c52db3b119b06e90ef8d03"} Nov 02 15:24:32.482957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.482934 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:32.609199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.609160 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca\") pod \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " Nov 02 15:24:32.609381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.609249 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q\" (UniqueName: \"kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q\") pod \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " Nov 02 15:24:32.609381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.609363 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca\") pod \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\" (UID: \"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c\") " Nov 02 15:24:32.609750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.609722 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca" pod "0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" (UID: "0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:32.610188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.610156 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:32.636596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.636559 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca" pod "0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" (UID: "0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:24:32.636724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.636606 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q" pod "0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" (UID: "0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c"). InnerVolumeSpecName "kube-api-access-56g4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:32.711947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.711861 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:32.711947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.711893 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56g4q\" (UniqueName: \"kubernetes.io/projected/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c-kube-api-access-56g4q\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:32.995085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:32.994982 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" podUID="c39978d8-1bfa-4bb4-a8fb-394c9b277d51" containerName="payload-pre-processing" containerID="cri-o://91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff" gracePeriod=25 Nov 02 15:24:33.282017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.281992 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:24:33.308443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.308408 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" Nov 02 15:24:33.308443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.308430 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz" event={"ID":"0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c","Type":"ContainerDied","Data":"f1a9f448e53f03fd034e89435a42b2f04fa11815219b64545d0e0ac4e0028745"} Nov 02 15:24:33.308740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.308489 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a9f448e53f03fd034e89435a42b2f04fa11815219b64545d0e0ac4e0028745" Nov 02 15:24:33.310164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.310137 5091 generic.go:356] "Generic (PLEG): container finished" podID="c39978d8-1bfa-4bb4-a8fb-394c9b277d51" containerID="91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff" exitCode=0 Nov 02 15:24:33.310293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.310182 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" Nov 02 15:24:33.310293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.310187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" event={"ID":"c39978d8-1bfa-4bb4-a8fb-394c9b277d51","Type":"ContainerDied","Data":"91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff"} Nov 02 15:24:33.310293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.310224 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522" event={"ID":"c39978d8-1bfa-4bb4-a8fb-394c9b277d51","Type":"ContainerDied","Data":"b243604d350086658f1ef3bf0a5f585a3016049d7b62710d2b170cca5fe9fd0d"} Nov 02 15:24:33.310293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.310256 5091 scope.go:122] "RemoveContainer" containerID="91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff" Nov 02 15:24:33.322656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.322568 5091 scope.go:122] "RemoveContainer" containerID="91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff" Nov 02 15:24:33.322909 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:24:33.322885 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff\": container with ID starting with 91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff not found: ID does not exist" containerID="91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff" Nov 02 15:24:33.322979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.322923 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff"} err="failed to get container status \"91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff\": rpc error: code = NotFound desc = could not find container \"91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff\": container with ID starting with 91c7d0d892a7a90a88138391c08c679da6ebfda5d1894f7cd5ebc220dc61a8ff not found: ID does not exist" Nov 02 15:24:33.424297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.424256 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume\") pod \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " Nov 02 15:24:33.424488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.424376 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss\" (UniqueName: \"kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss\") pod \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\" (UID: \"c39978d8-1bfa-4bb4-a8fb-394c9b277d51\") " Nov 02 15:24:33.424781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.424752 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume" pod "c39978d8-1bfa-4bb4-a8fb-394c9b277d51" (UID: "c39978d8-1bfa-4bb4-a8fb-394c9b277d51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:33.425268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.425242 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:33.427579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.427550 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss" pod "c39978d8-1bfa-4bb4-a8fb-394c9b277d51" (UID: "c39978d8-1bfa-4bb4-a8fb-394c9b277d51"). InnerVolumeSpecName "kube-api-access-cmgss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:33.526241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.526170 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmgss\" (UniqueName: \"kubernetes.io/projected/c39978d8-1bfa-4bb4-a8fb-394c9b277d51-kube-api-access-cmgss\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:33.730047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.729996 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522"] Nov 02 15:24:33.735010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.734979 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a758f7bf-d7c45557f-qk522"] Nov 02 15:24:33.962460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:33.962427 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39978d8-1bfa-4bb4-a8fb-394c9b277d51" path="/var/lib/kubelet/pods/c39978d8-1bfa-4bb4-a8fb-394c9b277d51/volumes" Nov 02 15:24:34.715608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.715566 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7"] Nov 02 15:24:34.716247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.716232 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c39978d8-1bfa-4bb4-a8fb-394c9b277d51" containerName="payload-pre-processing" Nov 02 15:24:34.716292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.716248 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39978d8-1bfa-4bb4-a8fb-394c9b277d51" containerName="payload-pre-processing" Nov 02 15:24:34.716292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.716270 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" containerName="revoke-keys" Nov 02 15:24:34.716292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.716275 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" containerName="revoke-keys" Nov 02 15:24:34.716402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.716393 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39978d8-1bfa-4bb4-a8fb-394c9b277d51" containerName="payload-pre-processing" Nov 02 15:24:34.716435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.716406 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" containerName="revoke-keys" Nov 02 15:24:34.721315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.721294 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.765246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.765210 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7"] Nov 02 15:24:34.805161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.805130 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:24:34.805325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.805247 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:24:34.849891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.849849 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.850086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.850069 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rv7\" (UniqueName: \"kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.850152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.850126 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.952296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.952255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.952510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.952428 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rv7\" (UniqueName: \"kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.952510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.952484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.953315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.953287 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:34.954933 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:34.954903 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:35.013534 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:35.013451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rv7\" (UniqueName: \"kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7\") pod \"maas-api-revoke-sub-c449ab27-x9ns7\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:35.037118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:35.037086 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:35.418939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:35.418909 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7"] Nov 02 15:24:35.421748 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:24:35.421721 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cefe491_1483_4905_9307_3bb2fb7dbbec.slice/crio-a7a8a4306e9bd2226daded993bd898273d49649a250320dd29d41ddce7d40b6a WatchSource:0}: Error finding container a7a8a4306e9bd2226daded993bd898273d49649a250320dd29d41ddce7d40b6a: Status 404 returned error can't find the container with id a7a8a4306e9bd2226daded993bd898273d49649a250320dd29d41ddce7d40b6a Nov 02 15:24:36.337328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:36.337299 5091 generic.go:356] "Generic (PLEG): container finished" podID="9cefe491-1483-4905-9307-3bb2fb7dbbec" containerID="0cd1019b2843cad714cbb855d8ed7bc382008ff5a485ecbe5ceeb6199b3d749e" exitCode=0 Nov 02 15:24:36.337774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:36.337387 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" event={"ID":"9cefe491-1483-4905-9307-3bb2fb7dbbec","Type":"ContainerDied","Data":"0cd1019b2843cad714cbb855d8ed7bc382008ff5a485ecbe5ceeb6199b3d749e"} Nov 02 15:24:36.337774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:36.337444 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" event={"ID":"9cefe491-1483-4905-9307-3bb2fb7dbbec","Type":"ContainerStarted","Data":"a7a8a4306e9bd2226daded993bd898273d49649a250320dd29d41ddce7d40b6a"} Nov 02 15:24:37.579396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.579370 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:37.593743 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.593709 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7\" (UniqueName: \"kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7\") pod \"9cefe491-1483-4905-9307-3bb2fb7dbbec\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " Nov 02 15:24:37.593872 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.593815 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca\") pod \"9cefe491-1483-4905-9307-3bb2fb7dbbec\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " Nov 02 15:24:37.594077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.594061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca\") pod \"9cefe491-1483-4905-9307-3bb2fb7dbbec\" (UID: \"9cefe491-1483-4905-9307-3bb2fb7dbbec\") " Nov 02 15:24:37.594181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.594157 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca" pod "9cefe491-1483-4905-9307-3bb2fb7dbbec" (UID: "9cefe491-1483-4905-9307-3bb2fb7dbbec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:37.594617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.594596 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:37.595932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.595907 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7" pod "9cefe491-1483-4905-9307-3bb2fb7dbbec" (UID: "9cefe491-1483-4905-9307-3bb2fb7dbbec"). InnerVolumeSpecName "kube-api-access-d7rv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:37.596009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.595978 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca" pod "9cefe491-1483-4905-9307-3bb2fb7dbbec" (UID: "9cefe491-1483-4905-9307-3bb2fb7dbbec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:24:37.695817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.695782 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cefe491-1483-4905-9307-3bb2fb7dbbec-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:37.695817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:37.695815 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7rv7\" (UniqueName: \"kubernetes.io/projected/9cefe491-1483-4905-9307-3bb2fb7dbbec-kube-api-access-d7rv7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:38.360563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:38.360449 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" Nov 02 15:24:38.360563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:38.360492 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7" event={"ID":"9cefe491-1483-4905-9307-3bb2fb7dbbec","Type":"ContainerDied","Data":"a7a8a4306e9bd2226daded993bd898273d49649a250320dd29d41ddce7d40b6a"} Nov 02 15:24:38.360769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:38.360567 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a8a4306e9bd2226daded993bd898273d49649a250320dd29d41ddce7d40b6a" Nov 02 15:24:46.064038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.064003 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb"] Nov 02 15:24:46.064711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.064694 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9cefe491-1483-4905-9307-3bb2fb7dbbec" containerName="revoke-keys" Nov 02 15:24:46.064711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.064711 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cefe491-1483-4905-9307-3bb2fb7dbbec" containerName="revoke-keys" Nov 02 15:24:46.064868 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.064858 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cefe491-1483-4905-9307-3bb2fb7dbbec" containerName="revoke-keys" Nov 02 15:24:46.068934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.068913 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.094357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.094325 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb"] Nov 02 15:24:46.119151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.119118 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:24:46.119271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.119235 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:24:46.157544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.151684 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.157912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.157872 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.158230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.158206 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkrrz\" (UniqueName: \"kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.259692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.259649 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.259903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.259848 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkrrz\" (UniqueName: \"kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.260083 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.260062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.260633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.260615 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.263117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.263090 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.320287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.320217 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkrrz\" (UniqueName: \"kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz\") pod \"maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.385329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.385294 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:46.820321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:46.820295 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb"] Nov 02 15:24:46.823769 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:24:46.823741 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5111f3ad_5fd2_478a_978b_948d31a72078.slice/crio-8ddd419619fb3eefa3e1aba7fe6cbaf25761bf8e7a234341fb083b4b5631a822 WatchSource:0}: Error finding container 8ddd419619fb3eefa3e1aba7fe6cbaf25761bf8e7a234341fb083b4b5631a822: Status 404 returned error can't find the container with id 8ddd419619fb3eefa3e1aba7fe6cbaf25761bf8e7a234341fb083b4b5631a822 Nov 02 15:24:47.483779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:47.483674 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" event={"ID":"5111f3ad-5fd2-478a-978b-948d31a72078","Type":"ContainerStarted","Data":"a8fdcd005a1d61561fae0959888579f923eed75509e21c4f9f61d93c5e6bad8a"} Nov 02 15:24:47.483779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:47.483753 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" event={"ID":"5111f3ad-5fd2-478a-978b-948d31a72078","Type":"ContainerStarted","Data":"8ddd419619fb3eefa3e1aba7fe6cbaf25761bf8e7a234341fb083b4b5631a822"} Nov 02 15:24:47.604420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:47.604352 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" podStartSLOduration=2.6043324930000002 podStartE2EDuration="2.604332493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:24:47.597906185 +0000 UTC m=+1641.762458742" watchObservedRunningTime="2026-11-02 15:24:47.604332493 +0000 UTC m=+1641.768885034" Nov 02 15:24:48.495816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:48.495782 5091 generic.go:356] "Generic (PLEG): container finished" podID="5111f3ad-5fd2-478a-978b-948d31a72078" containerID="a8fdcd005a1d61561fae0959888579f923eed75509e21c4f9f61d93c5e6bad8a" exitCode=0 Nov 02 15:24:48.496225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:48.495862 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" event={"ID":"5111f3ad-5fd2-478a-978b-948d31a72078","Type":"ContainerDied","Data":"a8fdcd005a1d61561fae0959888579f923eed75509e21c4f9f61d93c5e6bad8a"} Nov 02 15:24:49.725598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.725570 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:49.830927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.830831 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz\" (UniqueName: \"kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz\") pod \"5111f3ad-5fd2-478a-978b-948d31a72078\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " Nov 02 15:24:49.831097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.831037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca\") pod \"5111f3ad-5fd2-478a-978b-948d31a72078\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " Nov 02 15:24:49.831230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.831210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca\") pod \"5111f3ad-5fd2-478a-978b-948d31a72078\" (UID: \"5111f3ad-5fd2-478a-978b-948d31a72078\") " Nov 02 15:24:49.831506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.831453 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca" pod "5111f3ad-5fd2-478a-978b-948d31a72078" (UID: "5111f3ad-5fd2-478a-978b-948d31a72078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:24:49.831926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.831898 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:49.833123 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.833093 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz" pod "5111f3ad-5fd2-478a-978b-948d31a72078" (UID: "5111f3ad-5fd2-478a-978b-948d31a72078"). InnerVolumeSpecName "kube-api-access-dkrrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:24:49.833661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.833631 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca" pod "5111f3ad-5fd2-478a-978b-948d31a72078" (UID: "5111f3ad-5fd2-478a-978b-948d31a72078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:24:49.934091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.934044 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5111f3ad-5fd2-478a-978b-948d31a72078-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:49.934091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:49.934088 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkrrz\" (UniqueName: \"kubernetes.io/projected/5111f3ad-5fd2-478a-978b-948d31a72078-kube-api-access-dkrrz\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:24:50.514120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:50.514086 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" Nov 02 15:24:50.514120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:50.514108 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb" event={"ID":"5111f3ad-5fd2-478a-978b-948d31a72078","Type":"ContainerDied","Data":"8ddd419619fb3eefa3e1aba7fe6cbaf25761bf8e7a234341fb083b4b5631a822"} Nov 02 15:24:50.514357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:50.514159 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ddd419619fb3eefa3e1aba7fe6cbaf25761bf8e7a234341fb083b4b5631a822" Nov 02 15:24:51.523950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:51.523914 5091 generic.go:356] "Generic (PLEG): container finished" podID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerID="27efc52205b8263c43672f6c4737caa5f1e24cf596e9c2fbcdbbab21c0070d1a" exitCode=0 Nov 02 15:24:51.524408 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:51.523988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" event={"ID":"4c381744-eba9-4e6b-ae0a-c13e68c631ac","Type":"ContainerDied","Data":"27efc52205b8263c43672f6c4737caa5f1e24cf596e9c2fbcdbbab21c0070d1a"} Nov 02 15:24:51.524484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:51.524467 5091 scope.go:122] "RemoveContainer" containerID="27efc52205b8263c43672f6c4737caa5f1e24cf596e9c2fbcdbbab21c0070d1a" Nov 02 15:24:51.526179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:51.526146 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:51.526294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:51.526230 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:51.526294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:51.526279 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:24:52.604496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:52.604450 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" event={"ID":"4c381744-eba9-4e6b-ae0a-c13e68c631ac","Type":"ContainerStarted","Data":"a7f183f840d633cbb6101fa061559902647a0bbe9bc9322416108476bec52ca7"} Nov 02 15:24:52.818584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:52.818508 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:24:52.818893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:52.818871 5091 patch_prober.go:29] interesting pod/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 15:24:52.818946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:52.818926 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 15:24:53.819202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:53.819167 5091 patch_prober.go:29] interesting pod/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 15:24:53.819741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:53.819226 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 15:24:54.818484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:54.818445 5091 patch_prober.go:29] interesting pod/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 15:24:54.818693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:54.818505 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 15:24:55.818711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:55.818673 5091 patch_prober.go:29] interesting pod/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 15:24:55.819114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:55.818731 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 15:24:56.818388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:56.818355 5091 patch_prober.go:29] interesting pod/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 15:24:56.818565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:56.818411 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 15:24:57.622906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:57.622873 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9"] Nov 02 15:24:57.827683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:57.827648 5091 patch_prober.go:29] interesting pod/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 02 15:24:57.827954 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:57.827923 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 02 15:24:57.998155 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:24:57.998035 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:24:58.822782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:58.822746 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:24:58.823253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:58.823207 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:24:58.824076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:24:58.824051 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:25:02.623173 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.623114 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" podUID="c98ef18e-dc40-488b-98c2-5288176b6975" containerName="payload-pre-processing" containerID="cri-o://43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9" gracePeriod=25 Nov 02 15:25:02.931144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.931109 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:25:02.983606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.983576 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s\" (UniqueName: \"kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s\") pod \"c98ef18e-dc40-488b-98c2-5288176b6975\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " Nov 02 15:25:02.983800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.983746 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume\") pod \"c98ef18e-dc40-488b-98c2-5288176b6975\" (UID: \"c98ef18e-dc40-488b-98c2-5288176b6975\") " Nov 02 15:25:02.984174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.984149 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume" pod "c98ef18e-dc40-488b-98c2-5288176b6975" (UID: "c98ef18e-dc40-488b-98c2-5288176b6975"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:02.984546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.984506 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c98ef18e-dc40-488b-98c2-5288176b6975-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:02.985725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:02.985701 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s" pod "c98ef18e-dc40-488b-98c2-5288176b6975" (UID: "c98ef18e-dc40-488b-98c2-5288176b6975"). InnerVolumeSpecName "kube-api-access-cgd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:03.086891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.086854 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgd4s\" (UniqueName: \"kubernetes.io/projected/c98ef18e-dc40-488b-98c2-5288176b6975-kube-api-access-cgd4s\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:03.728179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.728138 5091 generic.go:356] "Generic (PLEG): container finished" podID="c98ef18e-dc40-488b-98c2-5288176b6975" containerID="43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9" exitCode=0 Nov 02 15:25:03.728676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.728201 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" Nov 02 15:25:03.728676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.728223 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" event={"ID":"c98ef18e-dc40-488b-98c2-5288176b6975","Type":"ContainerDied","Data":"43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9"} Nov 02 15:25:03.728676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.728326 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9" event={"ID":"c98ef18e-dc40-488b-98c2-5288176b6975","Type":"ContainerDied","Data":"c4f4c8b04eb1acf0d7ae0d5deed4550ab0fb0722d99e570e1351433939b5fdee"} Nov 02 15:25:03.728676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.728357 5091 scope.go:122] "RemoveContainer" containerID="43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9" Nov 02 15:25:03.741390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.741364 5091 scope.go:122] "RemoveContainer" containerID="43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9" Nov 02 15:25:03.741761 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:03.741730 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9\": container with ID starting with 43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9 not found: ID does not exist" containerID="43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9" Nov 02 15:25:03.741826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.741774 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9"} err="failed to get container status \"43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9\": rpc error: code = NotFound desc = could not find container \"43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9\": container with ID starting with 43edf194d644a98f984087d64e3fdcc76e7b49f5a63460e3349233f1c2d780e9 not found: ID does not exist" Nov 02 15:25:03.889375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.889331 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9"] Nov 02 15:25:03.935359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.935322 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-06a527d4-86cf6bbbc-7x5h9"] Nov 02 15:25:03.967852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:03.967815 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c98ef18e-dc40-488b-98c2-5288176b6975" path="/var/lib/kubelet/pods/c98ef18e-dc40-488b-98c2-5288176b6975/volumes" Nov 02 15:25:15.310550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:15.310459 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz"] Nov 02 15:25:15.375712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:15.375070 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a758f7bf-svdfz"] Nov 02 15:25:15.973795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:15.969019 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17381802-8e4f-4754-a2e1-69c517427016" path="/var/lib/kubelet/pods/17381802-8e4f-4754-a2e1-69c517427016/volumes" Nov 02 15:25:19.385681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.385648 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q"] Nov 02 15:25:19.386328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.386314 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5111f3ad-5fd2-478a-978b-948d31a72078" containerName="revoke-keys" Nov 02 15:25:19.386371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.386329 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5111f3ad-5fd2-478a-978b-948d31a72078" containerName="revoke-keys" Nov 02 15:25:19.386371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.386346 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c98ef18e-dc40-488b-98c2-5288176b6975" containerName="payload-pre-processing" Nov 02 15:25:19.386371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.386352 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c98ef18e-dc40-488b-98c2-5288176b6975" containerName="payload-pre-processing" Nov 02 15:25:19.386467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.386459 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5111f3ad-5fd2-478a-978b-948d31a72078" containerName="revoke-keys" Nov 02 15:25:19.386504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.386470 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c98ef18e-dc40-488b-98c2-5288176b6975" containerName="payload-pre-processing" Nov 02 15:25:19.390236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.390219 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.436904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.436873 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.437072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.437048 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.437123 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.437103 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tllf\" (UniqueName: \"kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.447622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.447596 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q"] Nov 02 15:25:19.486771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.486742 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:25:19.486771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.486747 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:25:19.538656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.538628 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tllf\" (UniqueName: \"kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.538792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.538780 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.538924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.538903 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.539453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.539432 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.541121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.541100 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.596889 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.596859 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tllf\" (UniqueName: \"kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf\") pod \"maas-api-revoke-keys-e2e-ait-89017f75-r499q\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:19.704211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:19.704133 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:20.086456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:20.086424 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q"] Nov 02 15:25:20.088435 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:25:20.088408 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49dde6e_e5cc_4960_b4c3_985bd98fcd51.slice/crio-e5fca0376148fda488d4cf7089a51fd04f70af26cfb1c3ee0de6b72c4a307e45 WatchSource:0}: Error finding container e5fca0376148fda488d4cf7089a51fd04f70af26cfb1c3ee0de6b72c4a307e45: Status 404 returned error can't find the container with id e5fca0376148fda488d4cf7089a51fd04f70af26cfb1c3ee0de6b72c4a307e45 Nov 02 15:25:20.964255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:20.964195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" event={"ID":"c49dde6e-e5cc-4960-b4c3-985bd98fcd51","Type":"ContainerStarted","Data":"ef5a343ed96e5bae94e64dbbb64914363bd3d7603b23a7d30b410400acf3229a"} Nov 02 15:25:20.964806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:20.964281 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" event={"ID":"c49dde6e-e5cc-4960-b4c3-985bd98fcd51","Type":"ContainerStarted","Data":"e5fca0376148fda488d4cf7089a51fd04f70af26cfb1c3ee0de6b72c4a307e45"} Nov 02 15:25:21.075417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:21.075339 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" podStartSLOduration=2.075324225 podStartE2EDuration="2.075324225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:21.074325097 +0000 UTC m=+1675.238877638" watchObservedRunningTime="2026-11-02 15:25:21.075324225 +0000 UTC m=+1675.239876765" Nov 02 15:25:21.974611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:21.974567 5091 generic.go:356] "Generic (PLEG): container finished" podID="c49dde6e-e5cc-4960-b4c3-985bd98fcd51" containerID="ef5a343ed96e5bae94e64dbbb64914363bd3d7603b23a7d30b410400acf3229a" exitCode=0 Nov 02 15:25:21.975095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:21.974665 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" event={"ID":"c49dde6e-e5cc-4960-b4c3-985bd98fcd51","Type":"ContainerDied","Data":"ef5a343ed96e5bae94e64dbbb64914363bd3d7603b23a7d30b410400acf3229a"} Nov 02 15:25:23.191289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.191257 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:23.295434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.295353 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca\") pod \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " Nov 02 15:25:23.295631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.295493 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca\") pod \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " Nov 02 15:25:23.295631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.295570 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf\" (UniqueName: \"kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf\") pod \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\" (UID: \"c49dde6e-e5cc-4960-b4c3-985bd98fcd51\") " Nov 02 15:25:23.295851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.295823 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca" pod "c49dde6e-e5cc-4960-b4c3-985bd98fcd51" (UID: "c49dde6e-e5cc-4960-b4c3-985bd98fcd51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:23.296291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.296271 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:23.297701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.297678 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca" pod "c49dde6e-e5cc-4960-b4c3-985bd98fcd51" (UID: "c49dde6e-e5cc-4960-b4c3-985bd98fcd51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:23.297795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.297725 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf" pod "c49dde6e-e5cc-4960-b4c3-985bd98fcd51" (UID: "c49dde6e-e5cc-4960-b4c3-985bd98fcd51"). InnerVolumeSpecName "kube-api-access-4tllf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:23.397999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.397962 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tllf\" (UniqueName: \"kubernetes.io/projected/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-kube-api-access-4tllf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:23.397999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.397992 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c49dde6e-e5cc-4960-b4c3-985bd98fcd51-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:23.992189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.992159 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" Nov 02 15:25:23.992348 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.992211 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q" event={"ID":"c49dde6e-e5cc-4960-b4c3-985bd98fcd51","Type":"ContainerDied","Data":"e5fca0376148fda488d4cf7089a51fd04f70af26cfb1c3ee0de6b72c4a307e45"} Nov 02 15:25:23.992348 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:23.992258 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5fca0376148fda488d4cf7089a51fd04f70af26cfb1c3ee0de6b72c4a307e45" Nov 02 15:25:25.607993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.607959 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf"] Nov 02 15:25:25.608859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.608835 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c49dde6e-e5cc-4960-b4c3-985bd98fcd51" containerName="revoke-keys" Nov 02 15:25:25.608859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.608858 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49dde6e-e5cc-4960-b4c3-985bd98fcd51" containerName="revoke-keys" Nov 02 15:25:25.609078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.609064 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49dde6e-e5cc-4960-b4c3-985bd98fcd51" containerName="revoke-keys" Nov 02 15:25:25.613037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.613017 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659628 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659597 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659657 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659712 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkchj\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659738 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659814 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659845 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.659923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.659907 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.660036 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.660023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.660071 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.660044 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.719134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.719089 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf"] Nov 02 15:25:25.761642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761607 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.761830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761673 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.761830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761694 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkchj\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.761830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761731 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.761830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761778 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.761830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761818 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.761913 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762065 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762090 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762062 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762091 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762182 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762379 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.762691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.762673 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.764381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.764353 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.764484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.764421 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.820174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.820142 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.820384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.820365 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkchj\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj\") pod \"maas-default-gateway-openshift-default-7f5667f86c-vwzpf\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:25.927565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:25.927529 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:26.379365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:26.379329 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf"] Nov 02 15:25:26.411010 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:25:26.410978 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25af89a0_678e_4c36_99dd_54000948f42e.slice/crio-2345eabda4d2987665cd1f728c302622db032b6ecd803028cd0ebd9d8700dcb4 WatchSource:0}: Error finding container 2345eabda4d2987665cd1f728c302622db032b6ecd803028cd0ebd9d8700dcb4: Status 404 returned error can't find the container with id 2345eabda4d2987665cd1f728c302622db032b6ecd803028cd0ebd9d8700dcb4 Nov 02 15:25:26.414468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:26.414432 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.414585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:26.414503 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:26.414585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:26.414560 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:25:27.030468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:27.030429 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" event={"ID":"25af89a0-678e-4c36-99dd-54000948f42e","Type":"ContainerStarted","Data":"bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c"} Nov 02 15:25:27.030468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:27.030501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" event={"ID":"25af89a0-678e-4c36-99dd-54000948f42e","Type":"ContainerStarted","Data":"2345eabda4d2987665cd1f728c302622db032b6ecd803028cd0ebd9d8700dcb4"} Nov 02 15:25:27.151616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:27.151565 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podStartSLOduration=2.151551091 podStartE2EDuration="2.151551091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:25:27.146672882 +0000 UTC m=+1681.311225435" watchObservedRunningTime="2026-11-02 15:25:27.151551091 +0000 UTC m=+1681.316103631" Nov 02 15:25:27.938416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:27.933199 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:27.946539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:27.940480 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:27.946539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:27.944644 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:27.979417 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:27.979311 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:25:28.665062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:28.663712 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz"] Nov 02 15:25:28.706660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:28.706596 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-bd9398b4-6fxkz"] Nov 02 15:25:28.712493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:28.709802 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q"] Nov 02 15:25:28.712493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:28.710034 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" containerID="cri-o://c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a" gracePeriod=30 Nov 02 15:25:28.947597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:28.947022 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:28.947597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:28.947099 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:29.932537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:29.931876 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:29.932537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:29.931947 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:29.963141 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:29.963092 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c" path="/var/lib/kubelet/pods/0b5b0265-500b-45bf-9fad-e7ffcf0fdf3c/volumes" Nov 02 15:25:30.930905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:30.930867 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:30.931210 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:30.931180 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:31.089861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:31.089825 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn"] Nov 02 15:25:31.934542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:31.930878 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:31.934542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:31.930949 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:32.936543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:32.934886 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:32.936543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:32.934951 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:33.928095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:33.928056 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:33.928268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:33.928117 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:34.040635 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.040611 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:25:34.082393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082367 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.082584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082467 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.082584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082506 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.082729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082618 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.082729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082679 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:34.082729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082726 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.082847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082771 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.082892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082853 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.083001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082884 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:34.083001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.082950 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.083114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083022 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs\") pod \"c1f21248-035b-4e39-9691-73f0b5a4e373\" (UID: \"c1f21248-035b-4e39-9691-73f0b5a4e373\") " Nov 02 15:25:34.083270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083248 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:34.083367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083334 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:34.083613 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083588 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:34.083945 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083918 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.084056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083950 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f21248-035b-4e39-9691-73f0b5a4e373-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.084056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083966 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.084056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083981 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.084056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.083996 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.084950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.084921 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "kube-api-access-tmrjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:34.085212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.085190 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:34.085336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.085318 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:25:34.085426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.085408 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token" pod "c1f21248-035b-4e39-9691-73f0b5a4e373" (UID: "c1f21248-035b-4e39-9691-73f0b5a4e373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:34.122822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.122792 5091 generic.go:356] "Generic (PLEG): container finished" podID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerID="c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a" exitCode=0 Nov 02 15:25:34.122940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.122853 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" Nov 02 15:25:34.122940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.122883 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" event={"ID":"c1f21248-035b-4e39-9691-73f0b5a4e373","Type":"ContainerDied","Data":"c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a"} Nov 02 15:25:34.123072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.122960 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q" event={"ID":"c1f21248-035b-4e39-9691-73f0b5a4e373","Type":"ContainerDied","Data":"9e1f68b745e1311b8636f68c11459e5da4697ebcf7920f6b6b8b3fbf1d8026ae"} Nov 02 15:25:34.123072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.122993 5091 scope.go:122] "RemoveContainer" containerID="c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a" Nov 02 15:25:34.134710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.134690 5091 scope.go:122] "RemoveContainer" containerID="2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18" Nov 02 15:25:34.147069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.147047 5091 scope.go:122] "RemoveContainer" containerID="c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a" Nov 02 15:25:34.147355 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:34.147334 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a\": container with ID starting with c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a not found: ID does not exist" containerID="c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a" Nov 02 15:25:34.147427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.147363 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a"} err="failed to get container status \"c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a\": rpc error: code = NotFound desc = could not find container \"c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a\": container with ID starting with c7c2325b9427dfaf604b5e76afcb1455d08617e589a16cd752cd39f90b531b9a not found: ID does not exist" Nov 02 15:25:34.147427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.147381 5091 scope.go:122] "RemoveContainer" containerID="2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18" Nov 02 15:25:34.147657 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:34.147638 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18\": container with ID starting with 2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18 not found: ID does not exist" containerID="2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18" Nov 02 15:25:34.147714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.147663 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18"} err="failed to get container status \"2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18\": rpc error: code = NotFound desc = could not find container \"2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18\": container with ID starting with 2ad4a88551e3780c3715ff35c73a923c586399774678911defc36fbe6c615e18 not found: ID does not exist" Nov 02 15:25:34.171648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.171622 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs"] Nov 02 15:25:34.171839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.171807 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" podUID="8646943b-1e40-4354-9d88-716d1abe31fd" containerName="maas-api" containerID="cri-o://eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d" gracePeriod=30 Nov 02 15:25:34.188451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.188420 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.188638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.188473 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.188638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.188503 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrjj\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-kube-api-access-tmrjj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.188638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.188550 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f21248-035b-4e39-9691-73f0b5a4e373-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.534908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.534817 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q"] Nov 02 15:25:34.550731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.550601 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bd9398b4-openshift-default-5fd859b665-4h75q"] Nov 02 15:25:34.599263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.598889 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:25:34.700978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.700942 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp\" (UniqueName: \"kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp\") pod \"8646943b-1e40-4354-9d88-716d1abe31fd\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " Nov 02 15:25:34.701358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.701331 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca\") pod \"8646943b-1e40-4354-9d88-716d1abe31fd\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " Nov 02 15:25:34.701731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.701699 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca" pod "8646943b-1e40-4354-9d88-716d1abe31fd" (UID: "8646943b-1e40-4354-9d88-716d1abe31fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:34.702023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.701991 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs\") pod \"8646943b-1e40-4354-9d88-716d1abe31fd\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " Nov 02 15:25:34.702345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.702330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls\") pod \"8646943b-1e40-4354-9d88-716d1abe31fd\" (UID: \"8646943b-1e40-4354-9d88-716d1abe31fd\") " Nov 02 15:25:34.703260 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.703241 5091 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8646943b-1e40-4354-9d88-716d1abe31fd-openshift-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.704230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.704203 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp" pod "8646943b-1e40-4354-9d88-716d1abe31fd" (UID: "8646943b-1e40-4354-9d88-716d1abe31fd"). InnerVolumeSpecName "kube-api-access-497mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:34.704990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.704951 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs" pod "8646943b-1e40-4354-9d88-716d1abe31fd" (UID: "8646943b-1e40-4354-9d88-716d1abe31fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:34.706996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.706965 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls" pod "8646943b-1e40-4354-9d88-716d1abe31fd" (UID: "8646943b-1e40-4354-9d88-716d1abe31fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:25:34.821570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.821391 5091 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-metrics-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.821570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.821432 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8646943b-1e40-4354-9d88-716d1abe31fd-maas-api-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.821570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.821450 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-497mp\" (UniqueName: \"kubernetes.io/projected/8646943b-1e40-4354-9d88-716d1abe31fd-kube-api-access-497mp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:34.928032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.927995 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 02 15:25:34.928257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:34.928071 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 02 15:25:35.137059 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.137023 5091 generic.go:356] "Generic (PLEG): container finished" podID="8646943b-1e40-4354-9d88-716d1abe31fd" containerID="eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d" exitCode=0 Nov 02 15:25:35.137561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.137085 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" Nov 02 15:25:35.137561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.137109 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" event={"ID":"8646943b-1e40-4354-9d88-716d1abe31fd","Type":"ContainerDied","Data":"eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d"} Nov 02 15:25:35.137561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.137178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs" event={"ID":"8646943b-1e40-4354-9d88-716d1abe31fd","Type":"ContainerDied","Data":"26154d9743d0eda5cf3ea1963e49b160bbb60d9ba2a716b05fb887b2b552933a"} Nov 02 15:25:35.137561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.137213 5091 scope.go:122] "RemoveContainer" containerID="eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d" Nov 02 15:25:35.148953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.148796 5091 scope.go:122] "RemoveContainer" containerID="eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d" Nov 02 15:25:35.149108 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:35.149089 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d\": container with ID starting with eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d not found: ID does not exist" containerID="eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d" Nov 02 15:25:35.149259 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.149235 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d"} err="failed to get container status \"eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d\": rpc error: code = NotFound desc = could not find container \"eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d\": container with ID starting with eed0000dbe33b162e773079645d91d6d8219069a3b6dd60a7de81cad5d58b50d not found: ID does not exist" Nov 02 15:25:35.312796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.312760 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs"] Nov 02 15:25:35.317223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.317201 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-06a527d4-64dcf7c8f7-lndrs"] Nov 02 15:25:35.935641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.935608 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:35.936460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.936427 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:35.936967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.936948 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:25:35.961589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.961556 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8646943b-1e40-4354-9d88-716d1abe31fd" path="/var/lib/kubelet/pods/8646943b-1e40-4354-9d88-716d1abe31fd/volumes" Nov 02 15:25:35.962083 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:35.962068 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" path="/var/lib/kubelet/pods/c1f21248-035b-4e39-9691-73f0b5a4e373/volumes" Nov 02 15:25:36.090481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.090444 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" podUID="b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" containerName="payload-processing" containerID="cri-o://b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470" gracePeriod=25 Nov 02 15:25:36.400944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.400916 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:25:36.549584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.548197 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv\" (UniqueName: \"kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv\") pod \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " Nov 02 15:25:36.549584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.548324 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume\") pod \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\" (UID: \"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5\") " Nov 02 15:25:36.549584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.549402 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume" pod "b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" (UID: "b8c7aaf7-0dee-4287-9686-1d5d26ad93d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:36.571809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.571777 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv" pod "b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" (UID: "b8c7aaf7-0dee-4287-9686-1d5d26ad93d5"). InnerVolumeSpecName "kube-api-access-dwqfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:36.650407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.650371 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqfv\" (UniqueName: \"kubernetes.io/projected/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-kube-api-access-dwqfv\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:36.650407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.650407 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:36.858090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.857974 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb"] Nov 02 15:25:36.865578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.865541 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-06a527d4-5gxlb"] Nov 02 15:25:36.915751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.915720 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck"] Nov 02 15:25:36.915989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:36.915964 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" containerID="cri-o://416b3e51a1b45a2423c2ef2f58597da61b4dc22c016f025c2edbb7d64e8b9d1c" gracePeriod=30 Nov 02 15:25:37.171539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.168850 5091 generic.go:356] "Generic (PLEG): container finished" podID="b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" containerID="b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470" exitCode=0 Nov 02 15:25:37.171539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.169281 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" event={"ID":"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5","Type":"ContainerDied","Data":"b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470"} Nov 02 15:25:37.171539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.169370 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" event={"ID":"b8c7aaf7-0dee-4287-9686-1d5d26ad93d5","Type":"ContainerDied","Data":"bd3966b9cac99a4c28968c77eafe1126c3f59ec766d1728a940b6a7cd12b9170"} Nov 02 15:25:37.171539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.169398 5091 scope.go:122] "RemoveContainer" containerID="b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470" Nov 02 15:25:37.171539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.170180 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn" Nov 02 15:25:37.189982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.189955 5091 scope.go:122] "RemoveContainer" containerID="b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470" Nov 02 15:25:37.190410 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:37.190390 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470\": container with ID starting with b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470 not found: ID does not exist" containerID="b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470" Nov 02 15:25:37.190596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.190417 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470"} err="failed to get container status \"b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470\": rpc error: code = NotFound desc = could not find container \"b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470\": container with ID starting with b0a9674fc6c491e17e2cf983dbd75aece4a73d826ac5a5bfaf73e6ba0b4b2470 not found: ID does not exist" Nov 02 15:25:37.340573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.340535 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn"] Nov 02 15:25:37.348168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.348140 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-89017f75-68d594d6f-q9qxn"] Nov 02 15:25:37.545117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.542159 5091 patch_prober.go:29] interesting pod/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 02 15:25:37.545117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.542237 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:25:37.966627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.966586 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5111f3ad-5fd2-478a-978b-948d31a72078" path="/var/lib/kubelet/pods/5111f3ad-5fd2-478a-978b-948d31a72078/volumes" Nov 02 15:25:37.967373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:37.967353 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" path="/var/lib/kubelet/pods/b8c7aaf7-0dee-4287-9686-1d5d26ad93d5/volumes" Nov 02 15:25:42.215055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.215013 5091 generic.go:356] "Generic (PLEG): container finished" podID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerID="416b3e51a1b45a2423c2ef2f58597da61b4dc22c016f025c2edbb7d64e8b9d1c" exitCode=0 Nov 02 15:25:42.215436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.215082 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" event={"ID":"166421cc-2839-41e7-8ac8-223db1bdc0de","Type":"ContainerDied","Data":"416b3e51a1b45a2423c2ef2f58597da61b4dc22c016f025c2edbb7d64e8b9d1c"} Nov 02 15:25:42.215436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.215159 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" event={"ID":"166421cc-2839-41e7-8ac8-223db1bdc0de","Type":"ContainerDied","Data":"3d1ad2ed3814f68573d88c6d7e3536cba1768bd38c72a61a4c89b394d2969366"} Nov 02 15:25:42.215436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.215182 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d1ad2ed3814f68573d88c6d7e3536cba1768bd38c72a61a4c89b394d2969366" Nov 02 15:25:42.215436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.215200 5091 scope.go:122] "RemoveContainer" containerID="b78050d8245551571d6eda65ea53e3386ec49d1954cb8e2194fe9598c12bcf08" Nov 02 15:25:42.221959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.221931 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:25:42.338240 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338208 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338394 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338433 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338468 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338565 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338665 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:42.338772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338702 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338745 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.338925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:42.338925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.338858 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data\") pod \"166421cc-2839-41e7-8ac8-223db1bdc0de\" (UID: \"166421cc-2839-41e7-8ac8-223db1bdc0de\") " Nov 02 15:25:42.339402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339077 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:42.339402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339240 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:25:42.339402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:42.339908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339874 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.339908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339903 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.340071 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339918 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.340071 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339931 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/166421cc-2839-41e7-8ac8-223db1bdc0de-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.340071 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.339944 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.341034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.340990 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:25:42.341300 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.341274 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "kube-api-access-6m2dp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:42.341423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.341332 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:25:42.341632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.341607 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token" pod "166421cc-2839-41e7-8ac8-223db1bdc0de" (UID: "166421cc-2839-41e7-8ac8-223db1bdc0de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:25:42.441880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.441793 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.441880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.441827 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.441880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.441841 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2dp\" (UniqueName: \"kubernetes.io/projected/166421cc-2839-41e7-8ac8-223db1bdc0de-kube-api-access-6m2dp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:42.441880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:42.441856 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/166421cc-2839-41e7-8ac8-223db1bdc0de-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:25:43.225305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:43.225274 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck" Nov 02 15:25:43.457054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:43.457005 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck"] Nov 02 15:25:43.461998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:43.461967 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-06a527d4-gw-openshift-default-6d9ccc6497-pd7ck"] Nov 02 15:25:43.965652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:25:43.965615 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" path="/var/lib/kubelet/pods/166421cc-2839-41e7-8ac8-223db1bdc0de/volumes" Nov 02 15:25:57.936268 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:25:57.936226 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:26:02.051864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:02.051826 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c"] Nov 02 15:26:02.098679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:02.098641 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7842d4ad-l555c"] Nov 02 15:26:03.963018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:03.962981 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="770968a3-5b11-4dc1-b1ef-9ad745d32c9a" path="/var/lib/kubelet/pods/770968a3-5b11-4dc1-b1ef-9ad745d32c9a/volumes" Nov 02 15:26:05.790054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:05.790019 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q"] Nov 02 15:26:05.836213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:05.836184 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-89017f75-r499q"] Nov 02 15:26:06.068778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:06.068676 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49dde6e-e5cc-4960-b4c3-985bd98fcd51" path="/var/lib/kubelet/pods/c49dde6e-e5cc-4960-b4c3-985bd98fcd51/volumes" Nov 02 15:26:06.381777 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:06.381746 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b"] Nov 02 15:26:06.382018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:06.381994 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" containerID="cri-o://a7f183f840d633cbb6101fa061559902647a0bbe9bc9322416108476bec52ca7" gracePeriod=30 Nov 02 15:26:11.609367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.609312 5091 generic.go:356] "Generic (PLEG): container finished" podID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerID="a7f183f840d633cbb6101fa061559902647a0bbe9bc9322416108476bec52ca7" exitCode=0 Nov 02 15:26:11.611952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.609399 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" event={"ID":"4c381744-eba9-4e6b-ae0a-c13e68c631ac","Type":"ContainerDied","Data":"a7f183f840d633cbb6101fa061559902647a0bbe9bc9322416108476bec52ca7"} Nov 02 15:26:11.611952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.610259 5091 scope.go:122] "RemoveContainer" containerID="27efc52205b8263c43672f6c4737caa5f1e24cf596e9c2fbcdbbab21c0070d1a" Nov 02 15:26:11.699729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.699701 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:26:11.896878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.896847 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.896894 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897014 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897086 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897130 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897152 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:11.897422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897212 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897275 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897337 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897377 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy\") pod \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\" (UID: \"4c381744-eba9-4e6b-ae0a-c13e68c631ac\") " Nov 02 15:26:11.897680 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:11.897741 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897712 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:26:11.898050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.897992 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:11.898050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.898008 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:11.898629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.898601 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:11.898739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.898634 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:11.898739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.898652 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:11.898739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.898667 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:11.898739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.898682 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:11.899630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.899593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "kube-api-access-599bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:11.899742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.899717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:26:11.899894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.899872 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:26:11.900239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:11.900219 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo" pod "4c381744-eba9-4e6b-ae0a-c13e68c631ac" (UID: "4c381744-eba9-4e6b-ae0a-c13e68c631ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:26:12.000386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.000357 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:12.000386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.000388 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:12.000564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.000398 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-599bj\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-kube-api-access-599bj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:12.000564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.000408 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c381744-eba9-4e6b-ae0a-c13e68c631ac-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:26:12.619560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.619503 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" Nov 02 15:26:12.619560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.619547 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b" event={"ID":"4c381744-eba9-4e6b-ae0a-c13e68c631ac","Type":"ContainerDied","Data":"56b96b0e54f97c11e9c13d678af5540232f5080e30e91f0dd70ed453a5ec02b6"} Nov 02 15:26:12.620015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.619620 5091 scope.go:122] "RemoveContainer" containerID="a7f183f840d633cbb6101fa061559902647a0bbe9bc9322416108476bec52ca7" Nov 02 15:26:12.830175 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.830139 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b"] Nov 02 15:26:12.879166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:12.879076 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-89017f75-openshift-default-55dfb48647-b5x5b"] Nov 02 15:26:13.962663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:13.962627 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" path="/var/lib/kubelet/pods/4c381744-eba9-4e6b-ae0a-c13e68c631ac/volumes" Nov 02 15:26:16.364161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.364123 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn"] Nov 02 15:26:16.365034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365010 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" containerName="payload-processing" Nov 02 15:26:16.365034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365032 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" containerName="payload-processing" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365049 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365057 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365067 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365075 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365087 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365094 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365106 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8646943b-1e40-4354-9d88-716d1abe31fd" containerName="maas-api" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365112 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8646943b-1e40-4354-9d88-716d1abe31fd" containerName="maas-api" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365139 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365146 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365178 5091 cpu_manager.go:414] "containerMap: removing container" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365186 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365199 5091 cpu_manager.go:414] "containerMap: removing container" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" Nov 02 15:26:16.365232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365207 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365377 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365395 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365405 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="166421cc-2839-41e7-8ac8-223db1bdc0de" containerName="istio-proxy" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365421 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c7aaf7-0dee-4287-9686-1d5d26ad93d5" containerName="payload-processing" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365430 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365441 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c381744-eba9-4e6b-ae0a-c13e68c631ac" containerName="istio-proxy" Nov 02 15:26:16.365927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.365450 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8646943b-1e40-4354-9d88-716d1abe31fd" containerName="maas-api" Nov 02 15:26:16.371550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.371530 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.460287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.460256 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d5b52693-openshift-default-dockercfg-qkms8\"" Nov 02 15:26:16.493081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.493047 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.493465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.493429 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.493763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.493743 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fmh\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.493898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.493884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.496599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.494016 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.523605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.500666 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.523926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.523882 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.524230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.524206 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.524496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.524476 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.601894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.601857 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn"] Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640488 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640570 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640652 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640721 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640818 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640940 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640943 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fmh\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.640994 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.641029 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.641112 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.641471 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.641506 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.641772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.642556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.642271 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.645061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.645034 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.646016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.645991 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.700831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.700801 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.715846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.704411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fmh\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh\") pod \"e2e-mt-d5b52693-openshift-default-7595b99869-947wn\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:16.762073 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:16.759817 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:17.245556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.245509 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn"] Nov 02 15:26:17.248740 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:26:17.248713 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5023c93f_ee49_49d8_90be_3897b8f104c4.slice/crio-28977dcb8ce33d78ff42011921150d37e18d51e134a06126766063b45766b0eb WatchSource:0}: Error finding container 28977dcb8ce33d78ff42011921150d37e18d51e134a06126766063b45766b0eb: Status 404 returned error can't find the container with id 28977dcb8ce33d78ff42011921150d37e18d51e134a06126766063b45766b0eb Nov 02 15:26:17.251485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.251447 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:17.251649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.251539 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:17.251649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.251586 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:26:17.672536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.672218 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" event={"ID":"5023c93f-ee49-49d8-90be-3897b8f104c4","Type":"ContainerStarted","Data":"da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46"} Nov 02 15:26:17.672536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.672461 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" event={"ID":"5023c93f-ee49-49d8-90be-3897b8f104c4","Type":"ContainerStarted","Data":"28977dcb8ce33d78ff42011921150d37e18d51e134a06126766063b45766b0eb"} Nov 02 15:26:17.760236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.760208 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:17.806186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:17.806115 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podStartSLOduration=1.806095735 podStartE2EDuration="1.806095735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:17.78836908 +0000 UTC m=+1731.952921621" watchObservedRunningTime="2026-11-02 15:26:17.806095735 +0000 UTC m=+1731.970648278" Nov 02 15:26:18.070138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:18.070039 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp"] Nov 02 15:26:18.075754 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:18.075711 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65c32e32-xf6cp"] Nov 02 15:26:18.767237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:18.767204 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:19.694544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:19.693193 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:19.699561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:19.698834 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:26:20.046588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:20.031560 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b183bed4-6983-409a-8f10-82d7ec35b33f" path="/var/lib/kubelet/pods/b183bed4-6983-409a-8f10-82d7ec35b33f/volumes" Nov 02 15:26:21.054265 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:21.054232 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf"] Nov 02 15:26:21.103944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:21.103908 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db90c09f-dspcf"] Nov 02 15:26:21.963580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:21.963537 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b678f54-4c83-4723-887d-ece04991a117" path="/var/lib/kubelet/pods/5b678f54-4c83-4723-887d-ece04991a117/volumes" Nov 02 15:26:24.431561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.431508 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd"] Nov 02 15:26:24.432943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.432918 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f21248-035b-4e39-9691-73f0b5a4e373" containerName="istio-proxy" Nov 02 15:26:24.437821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.437801 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.487045 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.487014 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd"] Nov 02 15:26:24.487633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.487608 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d5b52693\"" Nov 02 15:26:24.501720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.500874 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbn4\" (UniqueName: \"kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4\") pod \"payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.501720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.500966 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.532035 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.532005 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d5b52693-dockercfg-d824q\"" Nov 02 15:26:24.586608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.586576 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf"] Nov 02 15:26:24.591339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.591312 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:24.606201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.603301 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbn4\" (UniqueName: \"kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4\") pod \"payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.606201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.603375 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.606201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.604296 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.680541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.670946 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf"] Nov 02 15:26:24.730388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.729697 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbn4\" (UniqueName: \"kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4\") pod \"payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.731788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.730701 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4k5p\" (UniqueName: \"kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p\") pod \"payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:24.731788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.730854 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:24.755323 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.754964 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:24.847537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.846913 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4k5p\" (UniqueName: \"kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p\") pod \"payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:24.847537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.847238 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:24.851536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.848560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume\") pod \"payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:24.938011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:24.934867 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4k5p\" (UniqueName: \"kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p\") pod \"payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:25.211120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.211094 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd"] Nov 02 15:26:25.212570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.212545 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:25.214385 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:26:25.214358 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3667e3_1853_4323_b5e7_6f4132099f13.slice/crio-25a0250bdc4193c0dcd317231343bde0e3d4b878632343f7f37e55279efb819f WatchSource:0}: Error finding container 25a0250bdc4193c0dcd317231343bde0e3d4b878632343f7f37e55279efb819f: Status 404 returned error can't find the container with id 25a0250bdc4193c0dcd317231343bde0e3d4b878632343f7f37e55279efb819f Nov 02 15:26:25.599574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.599446 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf"] Nov 02 15:26:25.603017 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:26:25.602986 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cc8f3b_ce03_44ac_abc1_a87861c1005d.slice/crio-a395626c890bf1c2e91b720b0b5e2abe6fee15146225078f0f2f97ccc4f758f5 WatchSource:0}: Error finding container a395626c890bf1c2e91b720b0b5e2abe6fee15146225078f0f2f97ccc4f758f5: Status 404 returned error can't find the container with id a395626c890bf1c2e91b720b0b5e2abe6fee15146225078f0f2f97ccc4f758f5 Nov 02 15:26:25.752801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.752761 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" event={"ID":"0e3667e3-1853-4323-b5e7-6f4132099f13","Type":"ContainerStarted","Data":"2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715"} Nov 02 15:26:25.752972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.752835 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:25.752972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.752861 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" event={"ID":"0e3667e3-1853-4323-b5e7-6f4132099f13","Type":"ContainerStarted","Data":"25a0250bdc4193c0dcd317231343bde0e3d4b878632343f7f37e55279efb819f"} Nov 02 15:26:25.754833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.754803 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" event={"ID":"25cc8f3b-ce03-44ac-abc1-a87861c1005d","Type":"ContainerStarted","Data":"3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c"} Nov 02 15:26:25.754950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.754849 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" event={"ID":"25cc8f3b-ce03-44ac-abc1-a87861c1005d","Type":"ContainerStarted","Data":"a395626c890bf1c2e91b720b0b5e2abe6fee15146225078f0f2f97ccc4f758f5"} Nov 02 15:26:25.754950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.754876 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:25.869934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.869867 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" podStartSLOduration=1.869848369 podStartE2EDuration="1.869848369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:25.865942242 +0000 UTC m=+1740.030494783" watchObservedRunningTime="2026-11-02 15:26:25.869848369 +0000 UTC m=+1740.034400909" Nov 02 15:26:25.985926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:25.985863 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" podStartSLOduration=1.985844357 podStartE2EDuration="1.985844357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:25.975641983 +0000 UTC m=+1740.140194522" watchObservedRunningTime="2026-11-02 15:26:25.985844357 +0000 UTC m=+1740.150396899" Nov 02 15:26:27.936486 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:26:27.936449 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:26:29.586621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.586575 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d"] Nov 02 15:26:29.596007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.595972 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.636133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.636043 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pb5r\" (UniqueName: \"kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.636710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.636272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.643161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.643130 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d"] Nov 02 15:26:29.647447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.647427 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5aeac88e-dockercfg-m4wwh\"" Nov 02 15:26:29.647642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.647460 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5aeac88e\"" Nov 02 15:26:29.740593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.739434 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.740593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.739756 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pb5r\" (UniqueName: \"kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.741248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.741221 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.799366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.799327 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pb5r\" (UniqueName: \"kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r\") pod \"payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:29.912418 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:29.912379 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:30.391074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:30.391044 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d"] Nov 02 15:26:30.394099 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:26:30.394071 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b4a363_0c44_454a_8c6d_fd5b059d4890.slice/crio-e7009ce0d13f46ca44bbaeb6dc58e6bf84fc2dac1158402bb3fd0dd826d0a7e2 WatchSource:0}: Error finding container e7009ce0d13f46ca44bbaeb6dc58e6bf84fc2dac1158402bb3fd0dd826d0a7e2: Status 404 returned error can't find the container with id e7009ce0d13f46ca44bbaeb6dc58e6bf84fc2dac1158402bb3fd0dd826d0a7e2 Nov 02 15:26:30.809570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:30.809453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" event={"ID":"c0b4a363-0c44-454a-8c6d-fd5b059d4890","Type":"ContainerStarted","Data":"e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b"} Nov 02 15:26:30.809570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:30.809562 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" event={"ID":"c0b4a363-0c44-454a-8c6d-fd5b059d4890","Type":"ContainerStarted","Data":"e7009ce0d13f46ca44bbaeb6dc58e6bf84fc2dac1158402bb3fd0dd826d0a7e2"} Nov 02 15:26:30.809985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:30.809728 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:30.930763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:30.930695 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" podStartSLOduration=1.930674029 podStartE2EDuration="1.930674029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:30.92049088 +0000 UTC m=+1745.085043420" watchObservedRunningTime="2026-11-02 15:26:30.930674029 +0000 UTC m=+1745.095226575" Nov 02 15:26:36.765949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:36.765859 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:26:36.766395 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:36.766102 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:26:41.512616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.512583 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58"] Nov 02 15:26:41.517281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.517260 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.572531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.572481 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58"] Nov 02 15:26:41.618580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.613833 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7ad552\"" Nov 02 15:26:41.618835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.616480 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7ad552-dockercfg-ndh7m\"" Nov 02 15:26:41.645569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.643858 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.645569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.643916 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnd2s\" (UniqueName: \"kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s\") pod \"payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.708714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.708673 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf"] Nov 02 15:26:41.748769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.748224 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.748769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.748288 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnd2s\" (UniqueName: \"kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s\") pod \"payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.750541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.750067 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:41.752319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.752254 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.775171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.775129 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf"] Nov 02 15:26:41.816394 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.814618 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnd2s\" (UniqueName: \"kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s\") pod \"payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.822130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.822089 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:26:41.839580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.837853 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:41.851088 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.851054 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:41.851391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.851363 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxcs2\" (UniqueName: \"kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2\") pod \"payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:41.956869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.955381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxcs2\" (UniqueName: \"kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2\") pod \"payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:41.956869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.955601 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:41.957126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:41.957041 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:42.035927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.017794 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxcs2\" (UniqueName: \"kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2\") pod \"payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:42.080629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.079268 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:42.248597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.248565 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58"] Nov 02 15:26:42.509919 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.509887 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf"] Nov 02 15:26:42.511916 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:26:42.511886 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fb6960_de96_4379_adb3_0a014817beaa.slice/crio-0cb4ec3f6e5b73048da25200306e89652a10ebbcdc070d7f345a89d9e8a8dc5d WatchSource:0}: Error finding container 0cb4ec3f6e5b73048da25200306e89652a10ebbcdc070d7f345a89d9e8a8dc5d: Status 404 returned error can't find the container with id 0cb4ec3f6e5b73048da25200306e89652a10ebbcdc070d7f345a89d9e8a8dc5d Nov 02 15:26:42.981305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.981270 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" event={"ID":"aa336170-94a8-42dd-be65-8b294552805f","Type":"ContainerStarted","Data":"4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d"} Nov 02 15:26:42.981774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.981325 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" event={"ID":"aa336170-94a8-42dd-be65-8b294552805f","Type":"ContainerStarted","Data":"a358cb7645789ccf2acf791128c81253d136ba88f16c87331d18558ce7a4c136"} Nov 02 15:26:42.981774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.981352 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:42.982920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.982894 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" event={"ID":"62fb6960-de96-4379-adb3-0a014817beaa","Type":"ContainerStarted","Data":"b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792"} Nov 02 15:26:42.983034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.982938 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" event={"ID":"62fb6960-de96-4379-adb3-0a014817beaa","Type":"ContainerStarted","Data":"0cb4ec3f6e5b73048da25200306e89652a10ebbcdc070d7f345a89d9e8a8dc5d"} Nov 02 15:26:42.983034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:42.982985 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:43.097701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:43.097649 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" podStartSLOduration=2.097636629 podStartE2EDuration="2.097636629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:43.094666267 +0000 UTC m=+1757.259218807" watchObservedRunningTime="2026-11-02 15:26:43.097636629 +0000 UTC m=+1757.262189169" Nov 02 15:26:43.209638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:43.209571 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" podStartSLOduration=2.209553884 podStartE2EDuration="2.209553884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:26:43.205239553 +0000 UTC m=+1757.369792094" watchObservedRunningTime="2026-11-02 15:26:43.209553884 +0000 UTC m=+1757.374106427" Nov 02 15:26:50.049526 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:50.049482 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22"] Nov 02 15:26:50.097201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:50.097166 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81ef4479-v9g22"] Nov 02 15:26:51.961640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:51.961606 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3760e0-2421-4c88-a408-2200536074e1" path="/var/lib/kubelet/pods/7b3760e0-2421-4c88-a408-2200536074e1/volumes" Nov 02 15:26:53.993910 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:53.993875 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:26:53.994364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:53.993944 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:26:56.052335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:56.052303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685"] Nov 02 15:26:56.098858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:56.098828 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3336a14-hv685"] Nov 02 15:26:57.937057 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:26:57.937015 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:26:57.963678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:57.963646 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a4a8178-03f9-49b2-80d4-f4b7ce758c0b" path="/var/lib/kubelet/pods/3a4a8178-03f9-49b2-80d4-f4b7ce758c0b/volumes" Nov 02 15:26:59.058948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:59.058915 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg"] Nov 02 15:26:59.105485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:59.105452 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88f4867f-6tkkg"] Nov 02 15:26:59.963965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:26:59.963929 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fd3e6a-8a60-48ba-a682-659300a49c21" path="/var/lib/kubelet/pods/f6fd3e6a-8a60-48ba-a682-659300a49c21/volumes" Nov 02 15:27:10.096484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:10.096446 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9"] Nov 02 15:27:10.103393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:10.103364 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb8400db-69bn9"] Nov 02 15:27:11.962390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:11.962352 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a" path="/var/lib/kubelet/pods/b3f97404-f95f-4e66-b2b0-deb4c8ba7e7a/volumes" Nov 02 15:27:27.050068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:27.050033 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5"] Nov 02 15:27:27.096997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:27.096962 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0021fc6-lrdr5"] Nov 02 15:27:27.936813 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:27:27.936774 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:27:27.962019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:27.961986 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e938984-35f5-4011-a10b-b0c8ae2d72b7" path="/var/lib/kubelet/pods/5e938984-35f5-4011-a10b-b0c8ae2d72b7/volumes" Nov 02 15:27:28.580070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:28.580039 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:27:28.582684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:28.582661 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:27:28.593241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:28.593222 5091 kubelet.go:1689] "Image garbage collection succeeded" Nov 02 15:27:28.757863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:28.757829 5091 scope.go:122] "RemoveContainer" containerID="859cdee56dea98af8420afca866c77e2ac19cb94f5447532db21e33d60e321bd" Nov 02 15:27:28.771472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:28.771449 5091 scope.go:122] "RemoveContainer" containerID="1f169c68d4c5a9f51bc74411f7f7e703acd4c213c922bf39f29faff04e3ab8c4" Nov 02 15:27:28.781657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:28.781636 5091 scope.go:122] "RemoveContainer" containerID="18dbacdff221cf631b39676456c4c2a090d7a4196b8445c69922f0e6dbcd0c9d" Nov 02 15:27:35.049254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:35.049213 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5"] Nov 02 15:27:35.096676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:35.096643 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57d91221-wj8m5"] Nov 02 15:27:35.961423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:35.961379 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744" path="/var/lib/kubelet/pods/2bb6cbc1-a7ca-4d13-a26a-61e1c8cc3744/volumes" Nov 02 15:27:40.059198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:40.059160 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc"] Nov 02 15:27:40.104351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:40.104307 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ece7c605-ldqjc"] Nov 02 15:27:41.967633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:41.967595 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c86771-f35b-4a31-bcd6-16c06c9a6b8f" path="/var/lib/kubelet/pods/17c86771-f35b-4a31-bcd6-16c06c9a6b8f/volumes" Nov 02 15:27:44.607361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:44.607324 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94"] Nov 02 15:27:44.607890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:44.607557 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" containerID="cri-o://678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c" gracePeriod=30 Nov 02 15:27:46.051244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.051209 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk"] Nov 02 15:27:46.098004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.097961 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2698f4a9-nqdpk"] Nov 02 15:27:46.649961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.649932 5091 generic.go:356] "Generic (PLEG): container finished" podID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerID="da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46" exitCode=0 Nov 02 15:27:46.650178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.650005 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" event={"ID":"5023c93f-ee49-49d8-90be-3897b8f104c4","Type":"ContainerDied","Data":"da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46"} Nov 02 15:27:46.650401 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.650387 5091 scope.go:122] "RemoveContainer" containerID="da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46" Nov 02 15:27:46.651792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.651765 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:27:46.651905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.651829 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:27:46.651905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:46.651860 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:27:47.257543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.257107 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5"] Nov 02 15:27:47.257543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.257424 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" containerID="cri-o://375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59" gracePeriod=30 Nov 02 15:27:47.662703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.662666 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" event={"ID":"5023c93f-ee49-49d8-90be-3897b8f104c4","Type":"ContainerStarted","Data":"b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf"} Nov 02 15:27:47.760298 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.760262 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:27:47.762134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.762102 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:27:47.762267 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.762170 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 15:27:47.968892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:47.968804 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a33e49-9b35-47a8-a7f6-08d198062c1a" path="/var/lib/kubelet/pods/07a33e49-9b35-47a8-a7f6-08d198062c1a/volumes" Nov 02 15:27:48.760456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:48.760421 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:27:48.760896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:48.760473 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 15:27:49.693375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:49.693347 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:27:49.761662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:49.761623 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:27:49.762125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:49.761691 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 15:27:49.912020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:49.911998 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:27:50.004722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.004683 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.004953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.004739 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.004953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.004783 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.004953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.004814 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.005124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.004953 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.005124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005032 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.005124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005071 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.005124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005102 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.005334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005151 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.005334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo\") pod \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\" (UID: \"c15695c7-0a38-4dd0-9be0-1fae421eca1b\") " Nov 02 15:27:50.005334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005192 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.005334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005210 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.005552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005142 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.005552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005540 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:50.005907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005831 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.005907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005854 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.005907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005863 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.005907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005872 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.005907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.005881 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.007410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.007385 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:50.007714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.007690 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:50.007799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.007738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.007940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.007915 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt" pod "c15695c7-0a38-4dd0-9be0-1fae421eca1b" (UID: "c15695c7-0a38-4dd0-9be0-1fae421eca1b"). InnerVolumeSpecName "kube-api-access-hcmvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:50.107968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.107930 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.107968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.107967 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmvt\" (UniqueName: \"kubernetes.io/projected/c15695c7-0a38-4dd0-9be0-1fae421eca1b-kube-api-access-hcmvt\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.108196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.107983 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.108196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.107999 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c15695c7-0a38-4dd0-9be0-1fae421eca1b-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:50.695501 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.695461 5091 generic.go:356] "Generic (PLEG): container finished" podID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerID="678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c" exitCode=0 Nov 02 15:27:50.695715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.695557 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" Nov 02 15:27:50.695799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.695718 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" event={"ID":"c15695c7-0a38-4dd0-9be0-1fae421eca1b","Type":"ContainerDied","Data":"678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c"} Nov 02 15:27:50.695799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.695781 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94" event={"ID":"c15695c7-0a38-4dd0-9be0-1fae421eca1b","Type":"ContainerDied","Data":"e705fb68093235e87eaf32495a4264cd2dba90d65eae3673e206d0f8475c7156"} Nov 02 15:27:50.695908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.695801 5091 scope.go:122] "RemoveContainer" containerID="678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c" Nov 02 15:27:50.719545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.719502 5091 scope.go:122] "RemoveContainer" containerID="678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c" Nov 02 15:27:50.719918 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:27:50.719865 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c\": container with ID starting with 678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c not found: ID does not exist" containerID="678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c" Nov 02 15:27:50.719918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.719898 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c"} err="failed to get container status \"678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c\": rpc error: code = NotFound desc = could not find container \"678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c\": container with ID starting with 678a6c99f068ba759aa75201408d0d13acb4b025688883956ff407f503bb8a5c not found: ID does not exist" Nov 02 15:27:50.761013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.760981 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:27:50.761154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.761038 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 15:27:50.912031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.911995 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94"] Nov 02 15:27:50.920222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:50.920194 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-lwh94"] Nov 02 15:27:51.761007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:51.760970 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:27:51.761208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:51.761022 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 15:27:51.970155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:51.970121 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" path="/var/lib/kubelet/pods/c15695c7-0a38-4dd0-9be0-1fae421eca1b/volumes" Nov 02 15:27:52.636205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.636170 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:27:52.686430 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686396 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686680 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686619 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686695 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686715 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.686841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686756 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686835 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686874 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.686949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686938 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.687050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.686944 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.687144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687113 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket\") pod \"1f714a30-f65a-4712-a5ad-90015f83a450\" (UID: \"1f714a30-f65a-4712-a5ad-90015f83a450\") " Nov 02 15:27:52.687610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687453 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.687610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687469 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.687610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687580 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:27:52.687965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687936 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.688055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687972 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.688055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.687988 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.688055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.688001 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.688055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.688014 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f714a30-f65a-4712-a5ad-90015f83a450-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.689045 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.689006 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.689160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.689085 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "kube-api-access-kc8c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:27:52.689384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.689358 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:27:52.689861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.689840 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy" pod "1f714a30-f65a-4712-a5ad-90015f83a450" (UID: "1f714a30-f65a-4712-a5ad-90015f83a450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:27:52.717127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.717099 5091 generic.go:356] "Generic (PLEG): container finished" podID="1f714a30-f65a-4712-a5ad-90015f83a450" containerID="375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59" exitCode=0 Nov 02 15:27:52.717236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.717139 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" Nov 02 15:27:52.717236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.717179 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" event={"ID":"1f714a30-f65a-4712-a5ad-90015f83a450","Type":"ContainerDied","Data":"375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59"} Nov 02 15:27:52.717236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.717234 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5" event={"ID":"1f714a30-f65a-4712-a5ad-90015f83a450","Type":"ContainerDied","Data":"0ad526d61b54711e3656718cdbcd7e1403a62fa15d58873a536ce4509ff47122"} Nov 02 15:27:52.717385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.717256 5091 scope.go:122] "RemoveContainer" containerID="375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59" Nov 02 15:27:52.731154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.731135 5091 scope.go:122] "RemoveContainer" containerID="375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59" Nov 02 15:27:52.731497 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:27:52.731477 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59\": container with ID starting with 375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59 not found: ID does not exist" containerID="375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59" Nov 02 15:27:52.731592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.731512 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59"} err="failed to get container status \"375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59\": rpc error: code = NotFound desc = could not find container \"375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59\": container with ID starting with 375acd252c55ba1cd1dbd7c6e54d6cffd3280e12da353dd09e83e47833c33c59 not found: ID does not exist" Nov 02 15:27:52.761354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.761319 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 02 15:27:52.761502 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.761374 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 02 15:27:52.789201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.789150 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f714a30-f65a-4712-a5ad-90015f83a450-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.789201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.789192 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.789526 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.789209 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc8c8\" (UniqueName: \"kubernetes.io/projected/1f714a30-f65a-4712-a5ad-90015f83a450-kube-api-access-kc8c8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.789526 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.789222 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f714a30-f65a-4712-a5ad-90015f83a450-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:27:52.912964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.912933 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5"] Nov 02 15:27:52.917066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:52.917037 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-679p5"] Nov 02 15:27:53.766033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:53.766004 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:27:53.767260 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:53.767237 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:27:53.963075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:53.963040 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" path="/var/lib/kubelet/pods/1f714a30-f65a-4712-a5ad-90015f83a450/volumes" Nov 02 15:27:57.936374 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:27:57.936331 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:27:58.054920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:58.054887 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd"] Nov 02 15:27:58.098622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:58.098589 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8711a37-nh8zd"] Nov 02 15:27:59.965089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:27:59.965044 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efabd006-4322-41f3-870a-5214e56d34d7" path="/var/lib/kubelet/pods/efabd006-4322-41f3-870a-5214e56d34d7/volumes" Nov 02 15:28:04.139542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:04.130563 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl"] Nov 02 15:28:04.185541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:04.184593 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6eb75e8-f75gl"] Nov 02 15:28:05.961995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:05.961960 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b" path="/var/lib/kubelet/pods/c8fcef8d-473c-41bb-bdb2-e33e5d33ad2b/volumes" Nov 02 15:28:15.082958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:15.082874 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf"] Nov 02 15:28:15.119292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:15.119255 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f165ee86-h9sgf"] Nov 02 15:28:15.964448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:15.964410 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6512fd-d798-4863-ae51-2db0e27fad70" path="/var/lib/kubelet/pods/3b6512fd-d798-4863-ae51-2db0e27fad70/volumes" Nov 02 15:28:27.941262 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:28:27.941215 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:28:28.826165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:28.826136 5091 scope.go:122] "RemoveContainer" containerID="3fca94fd4ea3128b9f4c0df9dde3ae1f77cd7c38683add82a3f6663e1baed6da" Nov 02 15:28:28.839913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:28.839881 5091 scope.go:122] "RemoveContainer" containerID="da085605be7f799c983feebe258417885a10c67fba2ce0a4274c284cbf3c2058" Nov 02 15:28:28.850374 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:28.850347 5091 scope.go:122] "RemoveContainer" containerID="12ae423b874dbc1e036a8ce79888e7aabbb5b749db4a88d07cc1de63e91431a8" Nov 02 15:28:28.860341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:28.860317 5091 scope.go:122] "RemoveContainer" containerID="39d703ae6d780f7323fbe8dd34e4072810910f38ced3624e48c485f1a62a6c56" Nov 02 15:28:28.886410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:28.886386 5091 scope.go:122] "RemoveContainer" containerID="b5f82a28e416b0db97d4350db1a342ab0bd48d46b4ccb313c55a95f75cc824d5" Nov 02 15:28:46.691541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:46.690726 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf"] Nov 02 15:28:46.779016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:46.778981 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd"] Nov 02 15:28:51.691816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:51.691760 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" podUID="25cc8f3b-ce03-44ac-abc1-a87861c1005d" containerName="payload-processing" containerID="cri-o://3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c" gracePeriod=25 Nov 02 15:28:51.780109 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:51.780051 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" podUID="0e3667e3-1853-4323-b5e7-6f4132099f13" containerName="payload-pre-processing" containerID="cri-o://2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715" gracePeriod=25 Nov 02 15:28:52.105369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.105345 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:28:52.117949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.117926 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p\" (UniqueName: \"kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p\") pod \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " Nov 02 15:28:52.118138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.118122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume\") pod \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\" (UID: \"25cc8f3b-ce03-44ac-abc1-a87861c1005d\") " Nov 02 15:28:52.118588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.118562 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume" pod "25cc8f3b-ce03-44ac-abc1-a87861c1005d" (UID: "25cc8f3b-ce03-44ac-abc1-a87861c1005d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:28:52.119990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.119970 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p" pod "25cc8f3b-ce03-44ac-abc1-a87861c1005d" (UID: "25cc8f3b-ce03-44ac-abc1-a87861c1005d"). InnerVolumeSpecName "kube-api-access-z4k5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:28:52.139013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.138979 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:28:52.219191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.219120 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4\" (UniqueName: \"kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4\") pod \"0e3667e3-1853-4323-b5e7-6f4132099f13\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " Nov 02 15:28:52.219191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.219182 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume\") pod \"0e3667e3-1853-4323-b5e7-6f4132099f13\" (UID: \"0e3667e3-1853-4323-b5e7-6f4132099f13\") " Nov 02 15:28:52.219577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.219542 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume" pod "0e3667e3-1853-4323-b5e7-6f4132099f13" (UID: "0e3667e3-1853-4323-b5e7-6f4132099f13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:28:52.219701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.219688 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25cc8f3b-ce03-44ac-abc1-a87861c1005d-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:52.219751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.219706 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4k5p\" (UniqueName: \"kubernetes.io/projected/25cc8f3b-ce03-44ac-abc1-a87861c1005d-kube-api-access-z4k5p\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:52.219751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.219718 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e3667e3-1853-4323-b5e7-6f4132099f13-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:52.221015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.220994 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4" pod "0e3667e3-1853-4323-b5e7-6f4132099f13" (UID: "0e3667e3-1853-4323-b5e7-6f4132099f13"). InnerVolumeSpecName "kube-api-access-7cbn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:28:52.320789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.320755 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cbn4\" (UniqueName: \"kubernetes.io/projected/0e3667e3-1853-4323-b5e7-6f4132099f13-kube-api-access-7cbn4\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:28:52.380862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.380821 5091 generic.go:356] "Generic (PLEG): container finished" podID="0e3667e3-1853-4323-b5e7-6f4132099f13" containerID="2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715" exitCode=0 Nov 02 15:28:52.381127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.381095 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" event={"ID":"0e3667e3-1853-4323-b5e7-6f4132099f13","Type":"ContainerDied","Data":"2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715"} Nov 02 15:28:52.381311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.381161 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" event={"ID":"0e3667e3-1853-4323-b5e7-6f4132099f13","Type":"ContainerDied","Data":"25a0250bdc4193c0dcd317231343bde0e3d4b878632343f7f37e55279efb819f"} Nov 02 15:28:52.381311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.381193 5091 scope.go:122] "RemoveContainer" containerID="2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715" Nov 02 15:28:52.381445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.381362 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd" Nov 02 15:28:52.385609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.385576 5091 generic.go:356] "Generic (PLEG): container finished" podID="25cc8f3b-ce03-44ac-abc1-a87861c1005d" containerID="3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c" exitCode=0 Nov 02 15:28:52.385739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.385719 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" event={"ID":"25cc8f3b-ce03-44ac-abc1-a87861c1005d","Type":"ContainerDied","Data":"3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c"} Nov 02 15:28:52.385856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.385762 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" event={"ID":"25cc8f3b-ce03-44ac-abc1-a87861c1005d","Type":"ContainerDied","Data":"a395626c890bf1c2e91b720b0b5e2abe6fee15146225078f0f2f97ccc4f758f5"} Nov 02 15:28:52.385856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.385830 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf" Nov 02 15:28:52.399182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.399135 5091 scope.go:122] "RemoveContainer" containerID="2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715" Nov 02 15:28:52.399574 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:28:52.399548 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715\": container with ID starting with 2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715 not found: ID does not exist" containerID="2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715" Nov 02 15:28:52.399673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.399585 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715"} err="failed to get container status \"2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715\": rpc error: code = NotFound desc = could not find container \"2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715\": container with ID starting with 2c7845c8e267b454e42835ccd9039e2c49e2f924ea128921d78f62b2ca2d1715 not found: ID does not exist" Nov 02 15:28:52.399673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.399607 5091 scope.go:122] "RemoveContainer" containerID="3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c" Nov 02 15:28:52.414674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.413995 5091 scope.go:122] "RemoveContainer" containerID="3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c" Nov 02 15:28:52.414674 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:28:52.414255 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c\": container with ID starting with 3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c not found: ID does not exist" containerID="3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c" Nov 02 15:28:52.414674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.414282 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c"} err="failed to get container status \"3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c\": rpc error: code = NotFound desc = could not find container \"3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c\": container with ID starting with 3bb3f778b83cb27bb7c7d8ec357764f05c72b17335146c583a7640fae2774c1c not found: ID does not exist" Nov 02 15:28:52.573227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.573145 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd"] Nov 02 15:28:52.586816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.586780 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d5b52693-b4489b65f-hgpzd"] Nov 02 15:28:52.706303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.706272 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf"] Nov 02 15:28:52.718016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:52.717988 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d5b52693-d899fb4f9-gk7wf"] Nov 02 15:28:53.964751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:53.964714 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e3667e3-1853-4323-b5e7-6f4132099f13" path="/var/lib/kubelet/pods/0e3667e3-1853-4323-b5e7-6f4132099f13/volumes" Nov 02 15:28:53.965373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:28:53.965351 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cc8f3b-ce03-44ac-abc1-a87861c1005d" path="/var/lib/kubelet/pods/25cc8f3b-ce03-44ac-abc1-a87861c1005d/volumes" Nov 02 15:28:57.939935 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:28:57.939883 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:29:00.906391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:00.906354 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9"] Nov 02 15:29:00.906831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:00.906564 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" containerID="cri-o://07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb" gracePeriod=30 Nov 02 15:29:06.201982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.201956 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:29:06.257803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.257762 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.257959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.257817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.257959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.257935 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.258102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258082 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.258302 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.258452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258426 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.258585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258567 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.258690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258653 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.258752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258706 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs\") pod \"2217a43f-8407-41ac-a256-4e55c7786769\" (UID: \"2217a43f-8407-41ac-a256-4e55c7786769\") " Nov 02 15:29:06.259202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258135 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.259307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258323 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.259307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258493 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.259307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.258815 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:06.259307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.259101 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.259573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.259544 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.259647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.259577 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.259647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.259593 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2217a43f-8407-41ac-a256-4e55c7786769-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.259647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.259608 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.259647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.259623 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.260565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.260459 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.261121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.261093 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:06.261401 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.261371 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:06.261529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.261492 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl" pod "2217a43f-8407-41ac-a256-4e55c7786769" (UID: "2217a43f-8407-41ac-a256-4e55c7786769"). InnerVolumeSpecName "kube-api-access-bzrgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:06.361084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.361055 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2217a43f-8407-41ac-a256-4e55c7786769-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.361084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.361086 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrgl\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-kube-api-access-bzrgl\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.361266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.361097 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2217a43f-8407-41ac-a256-4e55c7786769-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.361266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.361106 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2217a43f-8407-41ac-a256-4e55c7786769-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:06.565055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.564954 5091 generic.go:356] "Generic (PLEG): container finished" podID="2217a43f-8407-41ac-a256-4e55c7786769" containerID="07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb" exitCode=0 Nov 02 15:29:06.565055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.565007 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" event={"ID":"2217a43f-8407-41ac-a256-4e55c7786769","Type":"ContainerDied","Data":"07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb"} Nov 02 15:29:06.565285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.565062 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" event={"ID":"2217a43f-8407-41ac-a256-4e55c7786769","Type":"ContainerDied","Data":"8ab62c26de042e5f0b33e91f72f65d016c5dc04487061b2c4811e181daac6fb7"} Nov 02 15:29:06.565285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.565093 5091 scope.go:122] "RemoveContainer" containerID="07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb" Nov 02 15:29:06.565285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.565273 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9" Nov 02 15:29:06.580697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.580669 5091 scope.go:122] "RemoveContainer" containerID="07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb" Nov 02 15:29:06.581058 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:29:06.581029 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb\": container with ID starting with 07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb not found: ID does not exist" containerID="07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb" Nov 02 15:29:06.581176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.581068 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb"} err="failed to get container status \"07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb\": rpc error: code = NotFound desc = could not find container \"07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb\": container with ID starting with 07417a985e03ce5208f2ffb79029d0723adf93e3c8c9b31a53f3b9f5545451cb not found: ID does not exist" Nov 02 15:29:06.739809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.739768 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9"] Nov 02 15:29:06.747436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:06.747410 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lmmc9"] Nov 02 15:29:07.964181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:07.964140 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2217a43f-8407-41ac-a256-4e55c7786769" path="/var/lib/kubelet/pods/2217a43f-8407-41ac-a256-4e55c7786769/volumes" Nov 02 15:29:23.406538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:23.406483 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn"] Nov 02 15:29:23.407364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:23.407326 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" containerID="cri-o://b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf" gracePeriod=30 Nov 02 15:29:23.768398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:23.768313 5091 patch_prober.go:29] interesting pod/e2e-mt-d5b52693-openshift-default-7595b99869-947wn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:29:23.768398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:23.768377 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:29:27.935646 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:29:27.935606 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:29:28.703547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.703486 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:29:28.804171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804136 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804256 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804332 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804494 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804598 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804634 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804688 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data\") pod \"5023c93f-ee49-49d8-90be-3897b8f104c4\" (UID: \"5023c93f-ee49-49d8-90be-3897b8f104c4\") " Nov 02 15:29:28.804947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804860 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:28.804947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804905 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:28.805193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.804958 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:28.805193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:28.805284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805216 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:29:28.805557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805531 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.805653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805560 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.805653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805575 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.805653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805589 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5023c93f-ee49-49d8-90be-3897b8f104c4-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.805653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.805602 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.806705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.806684 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:28.806970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.806947 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:29:28.807043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.806950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:29:28.807089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.807068 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh" pod "5023c93f-ee49-49d8-90be-3897b8f104c4" (UID: "5023c93f-ee49-49d8-90be-3897b8f104c4"). InnerVolumeSpecName "kube-api-access-g8fmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:29:28.816343 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.816322 5091 generic.go:356] "Generic (PLEG): container finished" podID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerID="b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf" exitCode=0 Nov 02 15:29:28.816434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.816370 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" Nov 02 15:29:28.816434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.816409 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" event={"ID":"5023c93f-ee49-49d8-90be-3897b8f104c4","Type":"ContainerDied","Data":"b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf"} Nov 02 15:29:28.816535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.816504 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn" event={"ID":"5023c93f-ee49-49d8-90be-3897b8f104c4","Type":"ContainerDied","Data":"28977dcb8ce33d78ff42011921150d37e18d51e134a06126766063b45766b0eb"} Nov 02 15:29:28.816604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.816564 5091 scope.go:122] "RemoveContainer" containerID="b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf" Nov 02 15:29:28.838476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.838459 5091 scope.go:122] "RemoveContainer" containerID="da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46" Nov 02 15:29:28.848040 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.848023 5091 scope.go:122] "RemoveContainer" containerID="b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf" Nov 02 15:29:28.848295 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:29:28.848267 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf\": container with ID starting with b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf not found: ID does not exist" containerID="b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf" Nov 02 15:29:28.848353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.848294 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf"} err="failed to get container status \"b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf\": rpc error: code = NotFound desc = could not find container \"b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf\": container with ID starting with b7592ece064cfa19f9b4e2f2fce82f9648c6f354dd51f0bb299e62536f71ebcf not found: ID does not exist" Nov 02 15:29:28.848353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.848315 5091 scope.go:122] "RemoveContainer" containerID="da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46" Nov 02 15:29:28.848569 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:29:28.848554 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46\": container with ID starting with da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46 not found: ID does not exist" containerID="da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46" Nov 02 15:29:28.848626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.848574 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46"} err="failed to get container status \"da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46\": rpc error: code = NotFound desc = could not find container \"da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46\": container with ID starting with da9ab62abf70f20c1e4fc4017bf19673135ae0ac802e3c3690d8dc0c9b9e0a46 not found: ID does not exist" Nov 02 15:29:28.906567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.906539 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fmh\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-kube-api-access-g8fmh\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.906567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.906567 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.906731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.906580 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.906731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.906589 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5023c93f-ee49-49d8-90be-3897b8f104c4-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:29:28.952338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.952251 5091 scope.go:122] "RemoveContainer" containerID="51ebefdb258471099a70f19dc6d9ffe736a67e051c0d288118e8d9cb6e9a6321" Nov 02 15:29:28.967171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.967149 5091 scope.go:122] "RemoveContainer" containerID="6ede590e0e69f10be3d40451c7e2610686cd1d4a70377b53a325700aa089a217" Nov 02 15:29:28.976775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.976755 5091 scope.go:122] "RemoveContainer" containerID="4e4528d845313dae5ac5195026b7fc246ea41b243c100fdc6dc702bd6b4010b1" Nov 02 15:29:28.986436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.986414 5091 scope.go:122] "RemoveContainer" containerID="615bb9e7ec4dd3bed701ddfe8e3857e3f120adf4694c1c5a6116b07479d897e8" Nov 02 15:29:28.990282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.990255 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn"] Nov 02 15:29:28.997877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.997856 5091 scope.go:122] "RemoveContainer" containerID="ffe2743bdd2ba06e95eb406e17b2313de9fac0d16e166e97bab9077b08c4ce4a" Nov 02 15:29:28.998081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:28.998062 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5b52693-openshift-default-7595b99869-947wn"] Nov 02 15:29:29.006464 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:29.006448 5091 scope.go:122] "RemoveContainer" containerID="8206974f2c36f24f6a3648df3655bec15539f9b7d2c606f3b207a8f40129f146" Nov 02 15:29:29.961815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:29.961785 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" path="/var/lib/kubelet/pods/5023c93f-ee49-49d8-90be-3897b8f104c4/volumes" Nov 02 15:29:38.053238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:38.053152 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7"] Nov 02 15:29:38.097766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:38.097737 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c449ab27-x9ns7"] Nov 02 15:29:39.963178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:29:39.963145 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cefe491-1483-4905-9307-3bb2fb7dbbec" path="/var/lib/kubelet/pods/9cefe491-1483-4905-9307-3bb2fb7dbbec/volumes" Nov 02 15:29:57.937043 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:29:57.937002 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:30:00.277886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.277850 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8"] Nov 02 15:30:00.278550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278500 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" Nov 02 15:30:00.278550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278530 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" Nov 02 15:30:00.278550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278549 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278555 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278571 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278577 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278590 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278595 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278601 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0e3667e3-1853-4323-b5e7-6f4132099f13" containerName="payload-pre-processing" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278606 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e3667e3-1853-4323-b5e7-6f4132099f13" containerName="payload-pre-processing" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278620 5091 cpu_manager.go:414] "containerMap: removing container" podUID="25cc8f3b-ce03-44ac-abc1-a87861c1005d" containerName="payload-processing" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278627 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cc8f3b-ce03-44ac-abc1-a87861c1005d" containerName="payload-processing" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278645 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" Nov 02 15:30:00.278688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278652 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278762 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2217a43f-8407-41ac-a256-4e55c7786769" containerName="istio-proxy" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278776 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e3667e3-1853-4323-b5e7-6f4132099f13" containerName="payload-pre-processing" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278795 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cc8f3b-ce03-44ac-abc1-a87861c1005d" containerName="payload-processing" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278802 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278811 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c15695c7-0a38-4dd0-9be0-1fae421eca1b" containerName="istio-proxy" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278819 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5023c93f-ee49-49d8-90be-3897b8f104c4" containerName="istio-proxy" Nov 02 15:30:00.279014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.278827 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f714a30-f65a-4712-a5ad-90015f83a450" containerName="istio-proxy" Nov 02 15:30:00.282482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.282462 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.296651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.296628 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt"] Nov 02 15:30:00.300652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.300638 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.352658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.352633 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb"] Nov 02 15:30:00.356760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.356743 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.382825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.382795 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8"] Nov 02 15:30:00.396122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.396100 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:30:00.407546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.404039 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7"] Nov 02 15:30:00.433503 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.433469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlwvx\" (UniqueName: \"kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.435597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.433773 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.435843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.435824 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7n9k\" (UniqueName: \"kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.435982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.435967 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.436662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.436640 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt"] Nov 02 15:30:00.436852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.436820 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s"] Nov 02 15:30:00.437183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.436752 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.442730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.442705 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h"] Nov 02 15:30:00.442895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.442878 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.458208 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.458182 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb"] Nov 02 15:30:00.458321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.458292 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.473444 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.473420 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7"] Nov 02 15:30:00.507601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.507576 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s"] Nov 02 15:30:00.510839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.510817 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h"] Nov 02 15:30:00.537874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.537808 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ld6\" (UniqueName: \"kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.538028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538004 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.538082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538065 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.538189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538171 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlwvx\" (UniqueName: \"kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.538256 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538241 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.538299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538285 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7n9k\" (UniqueName: \"kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.538353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.538437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdm8t\" (UniqueName: \"kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.538971 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.538951 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.539186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.539163 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.597867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.597835 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlwvx\" (UniqueName: \"kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.602904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.602883 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7n9k\" (UniqueName: \"kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k\") pod \"maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.613853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.613833 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:00.639896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.639866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkbc\" (UniqueName: \"kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc\") pod \"maas-api-key-cleanup-29893890-wf74h\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.640007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.639916 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdm8t\" (UniqueName: \"kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.640007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.639942 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-wf74h\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.640192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.640169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ld6\" (UniqueName: \"kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.640280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.640224 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.640280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.640272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs9r4\" (UniqueName: \"kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.640389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.640362 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.640446 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.640424 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.641097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.641074 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.641173 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.641128 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.728561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.726978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ld6\" (UniqueName: \"kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.728561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.727131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdm8t\" (UniqueName: \"kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t\") pod \"maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:00.741896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.741864 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkbc\" (UniqueName: \"kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc\") pod \"maas-api-key-cleanup-29893890-wf74h\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.742060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.741922 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-wf74h\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.742060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.742010 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.742174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.742069 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs9r4\" (UniqueName: \"kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.742730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.742701 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893890-wf74h\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.742831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.742819 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.776603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.776575 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:00.811630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.811602 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs9r4\" (UniqueName: \"kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4\") pod \"maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:00.811731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.811604 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkbc\" (UniqueName: \"kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc\") pod \"maas-api-key-cleanup-29893890-wf74h\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:00.896969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.896931 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:00.969767 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:00.969734 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:01.074350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.074173 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt"] Nov 02 15:30:01.077359 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:30:01.077328 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c180a15_d346_463e_9eaa_fd9b200c023e.slice/crio-23c6d08d82d6b9011e89e9b268bcc0cd36595c50b496758f468bbd267532ecf0 WatchSource:0}: Error finding container 23c6d08d82d6b9011e89e9b268bcc0cd36595c50b496758f468bbd267532ecf0: Status 404 returned error can't find the container with id 23c6d08d82d6b9011e89e9b268bcc0cd36595c50b496758f468bbd267532ecf0 Nov 02 15:30:01.088834 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.088808 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:01.099565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.099208 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:01.111195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.111117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" event={"ID":"6c180a15-d346-463e-9eaa-fd9b200c023e","Type":"ContainerStarted","Data":"23c6d08d82d6b9011e89e9b268bcc0cd36595c50b496758f468bbd267532ecf0"} Nov 02 15:30:01.173989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.173852 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7"] Nov 02 15:30:01.231320 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:30:01.231287 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c647a6_3311_48f4_83e1_045b5e2f0061.slice/crio-4534661e861e02c4f39015b8319f6198258b531942abbc5a0b5511df4212be9e WatchSource:0}: Error finding container 4534661e861e02c4f39015b8319f6198258b531942abbc5a0b5511df4212be9e: Status 404 returned error can't find the container with id 4534661e861e02c4f39015b8319f6198258b531942abbc5a0b5511df4212be9e Nov 02 15:30:01.367167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.367125 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8"] Nov 02 15:30:01.411453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.411423 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb"] Nov 02 15:30:01.415821 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:30:01.415792 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c65ab52_478d_4600_8421_a51ffacde722.slice/crio-2a0d531960004733458bf84a273b96f9c0a8a57d8242d1111f0fca553801f4bb WatchSource:0}: Error finding container 2a0d531960004733458bf84a273b96f9c0a8a57d8242d1111f0fca553801f4bb: Status 404 returned error can't find the container with id 2a0d531960004733458bf84a273b96f9c0a8a57d8242d1111f0fca553801f4bb Nov 02 15:30:01.590030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.590006 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s"] Nov 02 15:30:01.593010 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:30:01.592980 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338d94ff_8f6a_432e_8d20_5fc1d45bf837.slice/crio-8bf85e01eeb5279eca29920fb7c781d44034d7db1d0c53e0003352cae7f1878a WatchSource:0}: Error finding container 8bf85e01eeb5279eca29920fb7c781d44034d7db1d0c53e0003352cae7f1878a: Status 404 returned error can't find the container with id 8bf85e01eeb5279eca29920fb7c781d44034d7db1d0c53e0003352cae7f1878a Nov 02 15:30:01.597611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:01.597583 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h"] Nov 02 15:30:01.605435 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:30:01.605400 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e30f1cb_4d42_4c20_937b_a4022764d9f5.slice/crio-ea5d24645b29de0f31fa0140657f9180950411b90677e6008a40e2fec3f5264a WatchSource:0}: Error finding container ea5d24645b29de0f31fa0140657f9180950411b90677e6008a40e2fec3f5264a: Status 404 returned error can't find the container with id ea5d24645b29de0f31fa0140657f9180950411b90677e6008a40e2fec3f5264a Nov 02 15:30:02.121154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.121118 5091 generic.go:356] "Generic (PLEG): container finished" podID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerID="5c0ba24d65e3e8dc8c6a845ecd28abfad9baf54a73d378b53ad0a055650f3298" exitCode=60 Nov 02 15:30:02.121290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.121207 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" event={"ID":"6c180a15-d346-463e-9eaa-fd9b200c023e","Type":"ContainerDied","Data":"5c0ba24d65e3e8dc8c6a845ecd28abfad9baf54a73d378b53ad0a055650f3298"} Nov 02 15:30:02.121468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.121443 5091 scope.go:122] "RemoveContainer" containerID="5c0ba24d65e3e8dc8c6a845ecd28abfad9baf54a73d378b53ad0a055650f3298" Nov 02 15:30:02.122827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.122807 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerStarted","Data":"8bf85e01eeb5279eca29920fb7c781d44034d7db1d0c53e0003352cae7f1878a"} Nov 02 15:30:02.124596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.124568 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerStarted","Data":"0cffb1c0b95237ca58bc05527805f4a026df6a70d6a074f350f67e7e10658145"} Nov 02 15:30:02.124694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.124605 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerStarted","Data":"ea5d24645b29de0f31fa0140657f9180950411b90677e6008a40e2fec3f5264a"} Nov 02 15:30:02.126309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.126284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" event={"ID":"375810a9-ebb9-43e5-ab63-898a086490ba","Type":"ContainerStarted","Data":"a49a80dd49df7c9244f00d50bba01a58cbb6d8f87f67a51a17dfdc85a31604a7"} Nov 02 15:30:02.126402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.126329 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" event={"ID":"375810a9-ebb9-43e5-ab63-898a086490ba","Type":"ContainerStarted","Data":"3b4ef06cacb48f3ad8e4bc4b740fbd9c686af7db9779a27649eb55a9af0837c1"} Nov 02 15:30:02.128296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.128276 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerStarted","Data":"9f5ed1b32d4f1a183c10c2ec8e8cd4482dd405a066e6d54e1b4d614d63f27ee2"} Nov 02 15:30:02.128455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.128438 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerStarted","Data":"2a0d531960004733458bf84a273b96f9c0a8a57d8242d1111f0fca553801f4bb"} Nov 02 15:30:02.132918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.132897 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerStarted","Data":"90d8711ca003bbee18dac2497fb040bab8ba3042d1e8d04627dff84512b24dec"} Nov 02 15:30:02.133019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.132934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerStarted","Data":"4534661e861e02c4f39015b8319f6198258b531942abbc5a0b5511df4212be9e"} Nov 02 15:30:02.349434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.349389 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" podStartSLOduration=2.349376478 podStartE2EDuration="2.349376478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.348826692 +0000 UTC m=+1956.513379232" watchObservedRunningTime="2026-11-02 15:30:02.349376478 +0000 UTC m=+1956.513929010" Nov 02 15:30:02.460768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.460722 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" podStartSLOduration=2.460709041 podStartE2EDuration="2.460709041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.459935688 +0000 UTC m=+1956.624488228" watchObservedRunningTime="2026-11-02 15:30:02.460709041 +0000 UTC m=+1956.625261580" Nov 02 15:30:02.575365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.575317 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" podStartSLOduration=2.57530366 podStartE2EDuration="2.57530366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.573373645 +0000 UTC m=+1956.737926185" watchObservedRunningTime="2026-11-02 15:30:02.57530366 +0000 UTC m=+1956.739856237" Nov 02 15:30:02.689068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:02.688972 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" podStartSLOduration=2.688959045 podStartE2EDuration="2.688959045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:02.686192574 +0000 UTC m=+1956.850745128" watchObservedRunningTime="2026-11-02 15:30:02.688959045 +0000 UTC m=+1956.853511585" Nov 02 15:30:03.142415 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.142384 5091 generic.go:356] "Generic (PLEG): container finished" podID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerID="0cffb1c0b95237ca58bc05527805f4a026df6a70d6a074f350f67e7e10658145" exitCode=60 Nov 02 15:30:03.142566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.142429 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerDied","Data":"0cffb1c0b95237ca58bc05527805f4a026df6a70d6a074f350f67e7e10658145"} Nov 02 15:30:03.142810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.142793 5091 scope.go:122] "RemoveContainer" containerID="0cffb1c0b95237ca58bc05527805f4a026df6a70d6a074f350f67e7e10658145" Nov 02 15:30:03.144402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.144376 5091 generic.go:356] "Generic (PLEG): container finished" podID="375810a9-ebb9-43e5-ab63-898a086490ba" containerID="a49a80dd49df7c9244f00d50bba01a58cbb6d8f87f67a51a17dfdc85a31604a7" exitCode=60 Nov 02 15:30:03.144495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.144463 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" event={"ID":"375810a9-ebb9-43e5-ab63-898a086490ba","Type":"ContainerDied","Data":"a49a80dd49df7c9244f00d50bba01a58cbb6d8f87f67a51a17dfdc85a31604a7"} Nov 02 15:30:03.144713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.144696 5091 scope.go:122] "RemoveContainer" containerID="a49a80dd49df7c9244f00d50bba01a58cbb6d8f87f67a51a17dfdc85a31604a7" Nov 02 15:30:03.146511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.146368 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c65ab52-478d-4600-8421-a51ffacde722" containerID="9f5ed1b32d4f1a183c10c2ec8e8cd4482dd405a066e6d54e1b4d614d63f27ee2" exitCode=60 Nov 02 15:30:03.146511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.146460 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerDied","Data":"9f5ed1b32d4f1a183c10c2ec8e8cd4482dd405a066e6d54e1b4d614d63f27ee2"} Nov 02 15:30:03.146694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.146677 5091 scope.go:122] "RemoveContainer" containerID="9f5ed1b32d4f1a183c10c2ec8e8cd4482dd405a066e6d54e1b4d614d63f27ee2" Nov 02 15:30:03.148283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.148263 5091 generic.go:356] "Generic (PLEG): container finished" podID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerID="90d8711ca003bbee18dac2497fb040bab8ba3042d1e8d04627dff84512b24dec" exitCode=7 Nov 02 15:30:03.148376 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.148290 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerDied","Data":"90d8711ca003bbee18dac2497fb040bab8ba3042d1e8d04627dff84512b24dec"} Nov 02 15:30:03.148630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.148610 5091 scope.go:122] "RemoveContainer" containerID="90d8711ca003bbee18dac2497fb040bab8ba3042d1e8d04627dff84512b24dec" Nov 02 15:30:03.150468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.150447 5091 generic.go:356] "Generic (PLEG): container finished" podID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerID="dfee6ec473b57c2afcc62a1ab361e7206289a84abf5be1ca4980666a4a50d2fc" exitCode=60 Nov 02 15:30:03.150624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.150541 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" event={"ID":"6c180a15-d346-463e-9eaa-fd9b200c023e","Type":"ContainerDied","Data":"dfee6ec473b57c2afcc62a1ab361e7206289a84abf5be1ca4980666a4a50d2fc"} Nov 02 15:30:03.150624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.150604 5091 scope.go:122] "RemoveContainer" containerID="5c0ba24d65e3e8dc8c6a845ecd28abfad9baf54a73d378b53ad0a055650f3298" Nov 02 15:30:03.150805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.150786 5091 scope.go:122] "RemoveContainer" containerID="dfee6ec473b57c2afcc62a1ab361e7206289a84abf5be1ca4980666a4a50d2fc" Nov 02 15:30:03.151027 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:03.151000 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt_odh-ai-gateway-infra(6c180a15-d346-463e-9eaa-fd9b200c023e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" Nov 02 15:30:03.152142 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.152123 5091 generic.go:356] "Generic (PLEG): container finished" podID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerID="455ae51a410c8efae39b98f3dfbac8bf0e6266eb71f0bc37a3278680e6c78a01" exitCode=60 Nov 02 15:30:03.152227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.152200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerDied","Data":"455ae51a410c8efae39b98f3dfbac8bf0e6266eb71f0bc37a3278680e6c78a01"} Nov 02 15:30:03.152422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:03.152404 5091 scope.go:122] "RemoveContainer" containerID="455ae51a410c8efae39b98f3dfbac8bf0e6266eb71f0bc37a3278680e6c78a01" Nov 02 15:30:04.161735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.161707 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerStarted","Data":"5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c"} Nov 02 15:30:04.163680 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.163655 5091 generic.go:356] "Generic (PLEG): container finished" podID="375810a9-ebb9-43e5-ab63-898a086490ba" containerID="6a61e03bb3adddb731d39a41eb4722db9db5baeab604af72131cf5b3aa0b1f27" exitCode=60 Nov 02 15:30:04.163800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.163738 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" event={"ID":"375810a9-ebb9-43e5-ab63-898a086490ba","Type":"ContainerDied","Data":"6a61e03bb3adddb731d39a41eb4722db9db5baeab604af72131cf5b3aa0b1f27"} Nov 02 15:30:04.163800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.163791 5091 scope.go:122] "RemoveContainer" containerID="a49a80dd49df7c9244f00d50bba01a58cbb6d8f87f67a51a17dfdc85a31604a7" Nov 02 15:30:04.164061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.164045 5091 scope.go:122] "RemoveContainer" containerID="6a61e03bb3adddb731d39a41eb4722db9db5baeab604af72131cf5b3aa0b1f27" Nov 02 15:30:04.164418 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:04.164395 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8_odh-ai-gateway-infra(375810a9-ebb9-43e5-ab63-898a086490ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" Nov 02 15:30:04.166586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.166565 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerStarted","Data":"49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa"} Nov 02 15:30:04.168545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.168507 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerStarted","Data":"366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72"} Nov 02 15:30:04.170735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.170713 5091 scope.go:122] "RemoveContainer" containerID="dfee6ec473b57c2afcc62a1ab361e7206289a84abf5be1ca4980666a4a50d2fc" Nov 02 15:30:04.171053 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:04.171026 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt_odh-ai-gateway-infra(6c180a15-d346-463e-9eaa-fd9b200c023e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" Nov 02 15:30:04.172434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.172409 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerStarted","Data":"1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40"} Nov 02 15:30:04.882030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:04.881965 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" podStartSLOduration=4.881943274 podStartE2EDuration="4.881943274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:04.881852034 +0000 UTC m=+1959.046404574" watchObservedRunningTime="2026-11-02 15:30:04.881943274 +0000 UTC m=+1959.046495815" Nov 02 15:30:05.187894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.187807 5091 generic.go:356] "Generic (PLEG): container finished" podID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerID="5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c" exitCode=60 Nov 02 15:30:05.188335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.187886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerDied","Data":"5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c"} Nov 02 15:30:05.188335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.187946 5091 scope.go:122] "RemoveContainer" containerID="0cffb1c0b95237ca58bc05527805f4a026df6a70d6a074f350f67e7e10658145" Nov 02 15:30:05.188335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.188317 5091 scope.go:122] "RemoveContainer" containerID="5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c" Nov 02 15:30:05.188723 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:05.188695 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-wf74h_odh-ai-gateway-infra(2e30f1cb-4d42-4c20-937b-a4022764d9f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" Nov 02 15:30:05.190186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.190155 5091 scope.go:122] "RemoveContainer" containerID="6a61e03bb3adddb731d39a41eb4722db9db5baeab604af72131cf5b3aa0b1f27" Nov 02 15:30:05.190546 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:05.190489 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8_odh-ai-gateway-infra(375810a9-ebb9-43e5-ab63-898a086490ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" Nov 02 15:30:05.191812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.191791 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c65ab52-478d-4600-8421-a51ffacde722" containerID="49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa" exitCode=60 Nov 02 15:30:05.191905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.191874 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerDied","Data":"49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa"} Nov 02 15:30:05.192158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.192141 5091 scope.go:122] "RemoveContainer" containerID="49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa" Nov 02 15:30:05.192483 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:05.192455 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb_odh-ai-gateway-infra(1c65ab52-478d-4600-8421-a51ffacde722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" podUID="1c65ab52-478d-4600-8421-a51ffacde722" Nov 02 15:30:05.193845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.193822 5091 generic.go:356] "Generic (PLEG): container finished" podID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerID="366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72" exitCode=60 Nov 02 15:30:05.193926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.193907 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerDied","Data":"366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72"} Nov 02 15:30:05.194137 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.194120 5091 scope.go:122] "RemoveContainer" containerID="366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72" Nov 02 15:30:05.194395 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:05.194371 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7_odh-ai-gateway-infra(72c647a6-3311-48f4-83e1-045b5e2f0061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" Nov 02 15:30:05.195787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.195767 5091 generic.go:356] "Generic (PLEG): container finished" podID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerID="1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40" exitCode=60 Nov 02 15:30:05.195879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.195846 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerDied","Data":"1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40"} Nov 02 15:30:05.196037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.196023 5091 scope.go:122] "RemoveContainer" containerID="1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40" Nov 02 15:30:05.196285 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:05.196267 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s_odh-ai-gateway-infra(338d94ff-8f6a-432e-8d20-5fc1d45bf837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" Nov 02 15:30:05.201638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.201622 5091 scope.go:122] "RemoveContainer" containerID="9f5ed1b32d4f1a183c10c2ec8e8cd4482dd405a066e6d54e1b4d614d63f27ee2" Nov 02 15:30:05.214810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.214792 5091 scope.go:122] "RemoveContainer" containerID="90d8711ca003bbee18dac2497fb040bab8ba3042d1e8d04627dff84512b24dec" Nov 02 15:30:05.227711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:05.227690 5091 scope.go:122] "RemoveContainer" containerID="455ae51a410c8efae39b98f3dfbac8bf0e6266eb71f0bc37a3278680e6c78a01" Nov 02 15:30:06.206204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:06.206172 5091 scope.go:122] "RemoveContainer" containerID="366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72" Nov 02 15:30:06.206701 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:06.206617 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7_odh-ai-gateway-infra(72c647a6-3311-48f4-83e1-045b5e2f0061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" Nov 02 15:30:06.207892 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:06.207870 5091 scope.go:122] "RemoveContainer" containerID="1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40" Nov 02 15:30:06.208213 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:06.208190 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s_odh-ai-gateway-infra(338d94ff-8f6a-432e-8d20-5fc1d45bf837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" Nov 02 15:30:06.209625 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:06.209603 5091 scope.go:122] "RemoveContainer" containerID="5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c" Nov 02 15:30:06.209901 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:06.209872 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-wf74h_odh-ai-gateway-infra(2e30f1cb-4d42-4c20-937b-a4022764d9f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" Nov 02 15:30:06.211249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:06.211231 5091 scope.go:122] "RemoveContainer" containerID="49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa" Nov 02 15:30:06.211465 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:06.211447 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb_odh-ai-gateway-infra(1c65ab52-478d-4600-8421-a51ffacde722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" podUID="1c65ab52-478d-4600-8421-a51ffacde722" Nov 02 15:30:13.953047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:13.953011 5091 scope.go:122] "RemoveContainer" containerID="dfee6ec473b57c2afcc62a1ab361e7206289a84abf5be1ca4980666a4a50d2fc" Nov 02 15:30:14.277779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:14.277743 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" event={"ID":"6c180a15-d346-463e-9eaa-fd9b200c023e","Type":"ContainerStarted","Data":"ad627c5e6e4b4f99d3163997b03405053cd7ff67ddad5d6e947a3d4221ecee6e"} Nov 02 15:30:14.389664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:14.389611 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" podStartSLOduration=14.389597008 podStartE2EDuration="14.389597008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:30:14.388657291 +0000 UTC m=+1968.553209831" watchObservedRunningTime="2026-11-02 15:30:14.389597008 +0000 UTC m=+1968.554149547" Nov 02 15:30:15.286437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:15.286394 5091 generic.go:356] "Generic (PLEG): container finished" podID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerID="ad627c5e6e4b4f99d3163997b03405053cd7ff67ddad5d6e947a3d4221ecee6e" exitCode=60 Nov 02 15:30:15.286940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:15.286465 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" event={"ID":"6c180a15-d346-463e-9eaa-fd9b200c023e","Type":"ContainerDied","Data":"ad627c5e6e4b4f99d3163997b03405053cd7ff67ddad5d6e947a3d4221ecee6e"} Nov 02 15:30:15.286940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:15.286540 5091 scope.go:122] "RemoveContainer" containerID="dfee6ec473b57c2afcc62a1ab361e7206289a84abf5be1ca4980666a4a50d2fc" Nov 02 15:30:15.286940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:15.286905 5091 scope.go:122] "RemoveContainer" containerID="ad627c5e6e4b4f99d3163997b03405053cd7ff67ddad5d6e947a3d4221ecee6e" Nov 02 15:30:15.287257 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:15.287234 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt_odh-ai-gateway-infra(6c180a15-d346-463e-9eaa-fd9b200c023e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" Nov 02 15:30:15.409117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:15.409087 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt"] Nov 02 15:30:15.957133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:15.957086 5091 scope.go:122] "RemoveContainer" containerID="5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c" Nov 02 15:30:16.297314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.297280 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerStarted","Data":"6f686ec6f6bab6d91ce66cb0ac24ce26e45feb6fc9853b4e757710ebe5c22785"} Nov 02 15:30:16.496330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.496261 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:16.688290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.688257 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca\") pod \"6c180a15-d346-463e-9eaa-fd9b200c023e\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " Nov 02 15:30:16.688551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.688534 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx\" (UniqueName: \"kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx\") pod \"6c180a15-d346-463e-9eaa-fd9b200c023e\" (UID: \"6c180a15-d346-463e-9eaa-fd9b200c023e\") " Nov 02 15:30:16.688711 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.688687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca" pod "6c180a15-d346-463e-9eaa-fd9b200c023e" (UID: "6c180a15-d346-463e-9eaa-fd9b200c023e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:16.689124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.689097 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c180a15-d346-463e-9eaa-fd9b200c023e-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:16.692490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.692463 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx" pod "6c180a15-d346-463e-9eaa-fd9b200c023e" (UID: "6c180a15-d346-463e-9eaa-fd9b200c023e"). InnerVolumeSpecName "kube-api-access-zlwvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:16.791042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:16.790963 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlwvx\" (UniqueName: \"kubernetes.io/projected/6c180a15-d346-463e-9eaa-fd9b200c023e-kube-api-access-zlwvx\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:17.306395 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.306357 5091 generic.go:356] "Generic (PLEG): container finished" podID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerID="6f686ec6f6bab6d91ce66cb0ac24ce26e45feb6fc9853b4e757710ebe5c22785" exitCode=60 Nov 02 15:30:17.306879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.306431 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerDied","Data":"6f686ec6f6bab6d91ce66cb0ac24ce26e45feb6fc9853b4e757710ebe5c22785"} Nov 02 15:30:17.306879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.306535 5091 scope.go:122] "RemoveContainer" containerID="5c36ecd40842c7f4df009112f7c5a645471201f5d95c301bcb4409d76266fd1c" Nov 02 15:30:17.307013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.306876 5091 scope.go:122] "RemoveContainer" containerID="6f686ec6f6bab6d91ce66cb0ac24ce26e45feb6fc9853b4e757710ebe5c22785" Nov 02 15:30:17.307241 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:17.307219 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893890-wf74h_odh-ai-gateway-infra(2e30f1cb-4d42-4c20-937b-a4022764d9f5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" Nov 02 15:30:17.308100 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.308074 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" event={"ID":"6c180a15-d346-463e-9eaa-fd9b200c023e","Type":"ContainerDied","Data":"23c6d08d82d6b9011e89e9b268bcc0cd36595c50b496758f468bbd267532ecf0"} Nov 02 15:30:17.308166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.308117 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt" Nov 02 15:30:17.321048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.321031 5091 scope.go:122] "RemoveContainer" containerID="ad627c5e6e4b4f99d3163997b03405053cd7ff67ddad5d6e947a3d4221ecee6e" Nov 02 15:30:17.538482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.538452 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt"] Nov 02 15:30:17.544122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.544091 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7ad552-29893890-9hjdt"] Nov 02 15:30:17.554102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.554073 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h"] Nov 02 15:30:17.953245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.953206 5091 scope.go:122] "RemoveContainer" containerID="1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40" Nov 02 15:30:17.953561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.953540 5091 scope.go:122] "RemoveContainer" containerID="49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa" Nov 02 15:30:17.953671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.953646 5091 scope.go:122] "RemoveContainer" containerID="366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72" Nov 02 15:30:17.953739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.953722 5091 scope.go:122] "RemoveContainer" containerID="6a61e03bb3adddb731d39a41eb4722db9db5baeab604af72131cf5b3aa0b1f27" Nov 02 15:30:17.964161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:17.964137 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" path="/var/lib/kubelet/pods/6c180a15-d346-463e-9eaa-fd9b200c023e/volumes" Nov 02 15:30:18.319387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.319354 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerStarted","Data":"ae8ab70e113a28dffdb3682c34767e18e5b9a47d1a6098bdca1f73292d9f755f"} Nov 02 15:30:18.598809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.598784 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:18.617205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.617169 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca\") pod \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " Nov 02 15:30:18.617357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.617261 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc\" (UniqueName: \"kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc\") pod \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\" (UID: \"2e30f1cb-4d42-4c20-937b-a4022764d9f5\") " Nov 02 15:30:18.617495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.617471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca" pod "2e30f1cb-4d42-4c20-937b-a4022764d9f5" (UID: "2e30f1cb-4d42-4c20-937b-a4022764d9f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:18.617873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.617854 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e30f1cb-4d42-4c20-937b-a4022764d9f5-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:18.619325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.619299 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc" pod "2e30f1cb-4d42-4c20-937b-a4022764d9f5" (UID: "2e30f1cb-4d42-4c20-937b-a4022764d9f5"). InnerVolumeSpecName "kube-api-access-pmkbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:18.719379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:18.719340 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmkbc\" (UniqueName: \"kubernetes.io/projected/2e30f1cb-4d42-4c20-937b-a4022764d9f5-kube-api-access-pmkbc\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:19.330958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.330921 5091 generic.go:356] "Generic (PLEG): container finished" podID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerID="ae8ab70e113a28dffdb3682c34767e18e5b9a47d1a6098bdca1f73292d9f755f" exitCode=60 Nov 02 15:30:19.331440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.330991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerDied","Data":"ae8ab70e113a28dffdb3682c34767e18e5b9a47d1a6098bdca1f73292d9f755f"} Nov 02 15:30:19.331440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.331051 5091 scope.go:122] "RemoveContainer" containerID="1753c7d64d217a9e5e203e22a064932f6ce5909015370340c2cb7aa680caff40" Nov 02 15:30:19.331440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.331434 5091 scope.go:122] "RemoveContainer" containerID="ae8ab70e113a28dffdb3682c34767e18e5b9a47d1a6098bdca1f73292d9f755f" Nov 02 15:30:19.331794 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:19.331751 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s_odh-ai-gateway-infra(338d94ff-8f6a-432e-8d20-5fc1d45bf837)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" Nov 02 15:30:19.332944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.332922 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" Nov 02 15:30:19.333053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.332941 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h" event={"ID":"2e30f1cb-4d42-4c20-937b-a4022764d9f5","Type":"ContainerDied","Data":"ea5d24645b29de0f31fa0140657f9180950411b90677e6008a40e2fec3f5264a"} Nov 02 15:30:19.334852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.334831 5091 generic.go:356] "Generic (PLEG): container finished" podID="375810a9-ebb9-43e5-ab63-898a086490ba" containerID="4468e6c3e7c48950e33709b579deefc73ab1b95039119c30a62b0d0561952dea" exitCode=60 Nov 02 15:30:19.335191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.335167 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" event={"ID":"375810a9-ebb9-43e5-ab63-898a086490ba","Type":"ContainerDied","Data":"4468e6c3e7c48950e33709b579deefc73ab1b95039119c30a62b0d0561952dea"} Nov 02 15:30:19.335591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.335571 5091 scope.go:122] "RemoveContainer" containerID="4468e6c3e7c48950e33709b579deefc73ab1b95039119c30a62b0d0561952dea" Nov 02 15:30:19.335873 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:19.335848 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8_odh-ai-gateway-infra(375810a9-ebb9-43e5-ab63-898a086490ba)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" Nov 02 15:30:19.337075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.337052 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c65ab52-478d-4600-8421-a51ffacde722" containerID="98e00d95d94ab410287d0f28673bf5e293e87eaf88dbc1fd2c24769f1563d8dd" exitCode=60 Nov 02 15:30:19.337168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.337077 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerDied","Data":"98e00d95d94ab410287d0f28673bf5e293e87eaf88dbc1fd2c24769f1563d8dd"} Nov 02 15:30:19.337570 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.337431 5091 scope.go:122] "RemoveContainer" containerID="98e00d95d94ab410287d0f28673bf5e293e87eaf88dbc1fd2c24769f1563d8dd" Nov 02 15:30:19.337732 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:19.337712 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb_odh-ai-gateway-infra(1c65ab52-478d-4600-8421-a51ffacde722)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" podUID="1c65ab52-478d-4600-8421-a51ffacde722" Nov 02 15:30:19.339174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.339154 5091 generic.go:356] "Generic (PLEG): container finished" podID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerID="656e72d5973e69e38eec053b155005aa414f8d75a903e506c042d6e3406c7228" exitCode=60 Nov 02 15:30:19.339291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.339193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerDied","Data":"656e72d5973e69e38eec053b155005aa414f8d75a903e506c042d6e3406c7228"} Nov 02 15:30:19.339472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.339457 5091 scope.go:122] "RemoveContainer" containerID="656e72d5973e69e38eec053b155005aa414f8d75a903e506c042d6e3406c7228" Nov 02 15:30:19.339715 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:19.339695 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7_odh-ai-gateway-infra(72c647a6-3311-48f4-83e1-045b5e2f0061)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" Nov 02 15:30:19.346255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.346238 5091 scope.go:122] "RemoveContainer" containerID="6f686ec6f6bab6d91ce66cb0ac24ce26e45feb6fc9853b4e757710ebe5c22785" Nov 02 15:30:19.355370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.355349 5091 scope.go:122] "RemoveContainer" containerID="6a61e03bb3adddb731d39a41eb4722db9db5baeab604af72131cf5b3aa0b1f27" Nov 02 15:30:19.368360 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.368330 5091 scope.go:122] "RemoveContainer" containerID="49378c5c22a09ac569facbccd50b4096819d46f129edbe2e7b3080ddd9cc34fa" Nov 02 15:30:19.380802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.380783 5091 scope.go:122] "RemoveContainer" containerID="366b7301d01bd3f37122c666641aba03b9ff74d24ca34bfafbdb4f33a1f1ae72" Nov 02 15:30:19.445190 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.445160 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s"] Nov 02 15:30:19.826030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.825870 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h"] Nov 02 15:30:19.842830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.836706 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893890-wf74h"] Nov 02 15:30:19.961587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:19.961555 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" path="/var/lib/kubelet/pods/2e30f1cb-4d42-4c20-937b-a4022764d9f5/volumes" Nov 02 15:30:20.633441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.632565 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7"] Nov 02 15:30:20.705717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.705690 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:20.726846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.726819 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb"] Nov 02 15:30:20.800616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.800562 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca\") pod \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " Nov 02 15:30:20.800845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.800733 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4\" (UniqueName: \"kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4\") pod \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\" (UID: \"338d94ff-8f6a-432e-8d20-5fc1d45bf837\") " Nov 02 15:30:20.801158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.801117 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca" pod "338d94ff-8f6a-432e-8d20-5fc1d45bf837" (UID: "338d94ff-8f6a-432e-8d20-5fc1d45bf837"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:20.801549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.801468 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/338d94ff-8f6a-432e-8d20-5fc1d45bf837-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:20.805684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.805658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4" pod "338d94ff-8f6a-432e-8d20-5fc1d45bf837" (UID: "338d94ff-8f6a-432e-8d20-5fc1d45bf837"). InnerVolumeSpecName "kube-api-access-gs9r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:20.859682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.859651 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:20.904614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:20.903970 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs9r4\" (UniqueName: \"kubernetes.io/projected/338d94ff-8f6a-432e-8d20-5fc1d45bf837-kube-api-access-gs9r4\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.007554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.005969 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6\" (UniqueName: \"kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6\") pod \"72c647a6-3311-48f4-83e1-045b5e2f0061\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " Nov 02 15:30:21.007554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.006265 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca\") pod \"72c647a6-3311-48f4-83e1-045b5e2f0061\" (UID: \"72c647a6-3311-48f4-83e1-045b5e2f0061\") " Nov 02 15:30:21.011228 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.009869 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca" pod "72c647a6-3311-48f4-83e1-045b5e2f0061" (UID: "72c647a6-3311-48f4-83e1-045b5e2f0061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.011607 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.011577 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:21.033536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.026955 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6" pod "72c647a6-3311-48f4-83e1-045b5e2f0061" (UID: "72c647a6-3311-48f4-83e1-045b5e2f0061"). InnerVolumeSpecName "kube-api-access-k6ld6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.033536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.028588 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8"] Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.112197 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca\") pod \"1c65ab52-478d-4600-8421-a51ffacde722\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.112376 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t\" (UniqueName: \"kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t\") pod \"1c65ab52-478d-4600-8421-a51ffacde722\" (UID: \"1c65ab52-478d-4600-8421-a51ffacde722\") " Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.112907 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca" pod "1c65ab52-478d-4600-8421-a51ffacde722" (UID: "1c65ab52-478d-4600-8421-a51ffacde722"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.113357 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72c647a6-3311-48f4-83e1-045b5e2f0061-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.113382 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c65ab52-478d-4600-8421-a51ffacde722-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.113396 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6ld6\" (UniqueName: \"kubernetes.io/projected/72c647a6-3311-48f4-83e1-045b5e2f0061-kube-api-access-k6ld6\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.121548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.116597 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t" pod "1c65ab52-478d-4600-8421-a51ffacde722" (UID: "1c65ab52-478d-4600-8421-a51ffacde722"). InnerVolumeSpecName "kube-api-access-fdm8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.215568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.215285 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdm8t\" (UniqueName: \"kubernetes.io/projected/1c65ab52-478d-4600-8421-a51ffacde722-kube-api-access-fdm8t\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.257923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.257895 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:21.315999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.315961 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca\") pod \"375810a9-ebb9-43e5-ab63-898a086490ba\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " Nov 02 15:30:21.316164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.316133 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k\" (UniqueName: \"kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k\") pod \"375810a9-ebb9-43e5-ab63-898a086490ba\" (UID: \"375810a9-ebb9-43e5-ab63-898a086490ba\") " Nov 02 15:30:21.316317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.316296 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca" pod "375810a9-ebb9-43e5-ab63-898a086490ba" (UID: "375810a9-ebb9-43e5-ab63-898a086490ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:21.316652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.316633 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/375810a9-ebb9-43e5-ab63-898a086490ba-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.318214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.318190 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k" pod "375810a9-ebb9-43e5-ab63-898a086490ba" (UID: "375810a9-ebb9-43e5-ab63-898a086490ba"). InnerVolumeSpecName "kube-api-access-t7n9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:21.382328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.382296 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" Nov 02 15:30:21.382564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.382297 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s" event={"ID":"338d94ff-8f6a-432e-8d20-5fc1d45bf837","Type":"ContainerDied","Data":"8bf85e01eeb5279eca29920fb7c781d44034d7db1d0c53e0003352cae7f1878a"} Nov 02 15:30:21.382564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.382444 5091 scope.go:122] "RemoveContainer" containerID="ae8ab70e113a28dffdb3682c34767e18e5b9a47d1a6098bdca1f73292d9f755f" Nov 02 15:30:21.384233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.384211 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" Nov 02 15:30:21.384233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.384219 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8" event={"ID":"375810a9-ebb9-43e5-ab63-898a086490ba","Type":"ContainerDied","Data":"3b4ef06cacb48f3ad8e4bc4b740fbd9c686af7db9779a27649eb55a9af0837c1"} Nov 02 15:30:21.385758 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.385737 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" Nov 02 15:30:21.385758 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.385742 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb" event={"ID":"1c65ab52-478d-4600-8421-a51ffacde722","Type":"ContainerDied","Data":"2a0d531960004733458bf84a273b96f9c0a8a57d8242d1111f0fca553801f4bb"} Nov 02 15:30:21.387360 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.387334 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" event={"ID":"72c647a6-3311-48f4-83e1-045b5e2f0061","Type":"ContainerDied","Data":"4534661e861e02c4f39015b8319f6198258b531942abbc5a0b5511df4212be9e"} Nov 02 15:30:21.387473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.387361 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7" Nov 02 15:30:21.399657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.399634 5091 scope.go:122] "RemoveContainer" containerID="4468e6c3e7c48950e33709b579deefc73ab1b95039119c30a62b0d0561952dea" Nov 02 15:30:21.410414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.410397 5091 scope.go:122] "RemoveContainer" containerID="98e00d95d94ab410287d0f28673bf5e293e87eaf88dbc1fd2c24769f1563d8dd" Nov 02 15:30:21.418137 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.418113 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7n9k\" (UniqueName: \"kubernetes.io/projected/375810a9-ebb9-43e5-ab63-898a086490ba-kube-api-access-t7n9k\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:21.421196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.421174 5091 scope.go:122] "RemoveContainer" containerID="656e72d5973e69e38eec053b155005aa414f8d75a903e506c042d6e3406c7228" Nov 02 15:30:21.535031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.534952 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8"] Nov 02 15:30:21.538894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.538860 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-891723-29893890-ssxq8"] Nov 02 15:30:21.643434 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.643369 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7"] Nov 02 15:30:21.647787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.647760 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ca6972-29893890-gsgk7"] Nov 02 15:30:21.767047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.766767 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s"] Nov 02 15:30:21.772941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.772909 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-71d1eb-29893890-6858s"] Nov 02 15:30:21.876648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.876617 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb"] Nov 02 15:30:21.880235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.880211 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-61a835-29893890-rwlmb"] Nov 02 15:30:21.962854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.962817 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c65ab52-478d-4600-8421-a51ffacde722" path="/var/lib/kubelet/pods/1c65ab52-478d-4600-8421-a51ffacde722/volumes" Nov 02 15:30:21.963378 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.963357 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" path="/var/lib/kubelet/pods/338d94ff-8f6a-432e-8d20-5fc1d45bf837/volumes" Nov 02 15:30:21.963827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.963810 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" path="/var/lib/kubelet/pods/375810a9-ebb9-43e5-ab63-898a086490ba/volumes" Nov 02 15:30:21.964500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:21.964483 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" path="/var/lib/kubelet/pods/72c647a6-3311-48f4-83e1-045b5e2f0061/volumes" Nov 02 15:30:27.937075 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:27.937028 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:30:29.077493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:29.077460 5091 scope.go:122] "RemoveContainer" containerID="41b2df95b6a7604fe6ccd7d34499e81532fcbe387767ffc8ea8431dfa136a862" Nov 02 15:30:29.088069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:29.088049 5091 scope.go:122] "RemoveContainer" containerID="416b3e51a1b45a2423c2ef2f58597da61b4dc22c016f025c2edbb7d64e8b9d1c" Nov 02 15:30:45.167682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:45.167646 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj"] Nov 02 15:30:45.168176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:45.167833 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" containerID="cri-o://a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46" gracePeriod=30 Nov 02 15:30:50.549716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.549427 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:30:50.564752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.564712 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.564752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.564763 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.564752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.564799 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.564752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.564838 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.565281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.564879 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.565281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.564945 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.565281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.565037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.565281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.565157 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.565281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.565267 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data\") pod \"514400cc-4295-410f-917c-3437fc726399\" (UID: \"514400cc-4295-410f-917c-3437fc726399\") " Nov 02 15:30:50.566372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.566345 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:50.566834 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.566806 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:50.567050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.567027 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:50.567428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.567394 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:50.567551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.567478 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:30:50.569868 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.569842 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:50.570706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.570672 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:30:50.579025 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.578981 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:30:50.593828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.593762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r" pod "514400cc-4295-410f-917c-3437fc726399" (UID: "514400cc-4295-410f-917c-3437fc726399"). InnerVolumeSpecName "kube-api-access-qhp7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:30:50.661411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.661374 5091 generic.go:356] "Generic (PLEG): container finished" podID="514400cc-4295-410f-917c-3437fc726399" containerID="a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46" exitCode=0 Nov 02 15:30:50.661624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.661425 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" Nov 02 15:30:50.661624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.661454 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" event={"ID":"514400cc-4295-410f-917c-3437fc726399","Type":"ContainerDied","Data":"a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46"} Nov 02 15:30:50.661624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.661566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" event={"ID":"514400cc-4295-410f-917c-3437fc726399","Type":"ContainerDied","Data":"03aa4485bdff735bb83771a491fa09280e2007be16be197427780fa94da91285"} Nov 02 15:30:50.661624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.661602 5091 scope.go:122] "RemoveContainer" containerID="a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46" Nov 02 15:30:50.667051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667023 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/514400cc-4295-410f-917c-3437fc726399-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667055 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667070 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667082 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667097 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhp7r\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-kube-api-access-qhp7r\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667111 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667135 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/514400cc-4295-410f-917c-3437fc726399-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667148 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/514400cc-4295-410f-917c-3437fc726399-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.667177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.667161 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/514400cc-4295-410f-917c-3437fc726399-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:30:50.674231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.674213 5091 scope.go:122] "RemoveContainer" containerID="a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46" Nov 02 15:30:50.674558 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:50.674535 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46\": container with ID starting with a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46 not found: ID does not exist" containerID="a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46" Nov 02 15:30:50.674656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.674565 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46"} err="failed to get container status \"a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46\": rpc error: code = NotFound desc = could not find container \"a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46\": container with ID starting with a6f9584a3418202166c46ed41f6277309204a5f7ae7f42a82aa49234ff005d46 not found: ID does not exist" Nov 02 15:30:50.823867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.823746 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj"] Nov 02 15:30:50.832329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:50.832296 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj"] Nov 02 15:30:51.415540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:51.410899 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:30:51.415540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:51.410975 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-fkdzj" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:30:51.963564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:30:51.963510 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514400cc-4295-410f-917c-3437fc726399" path="/var/lib/kubelet/pods/514400cc-4295-410f-917c-3437fc726399/volumes" Nov 02 15:30:57.935863 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:30:57.935821 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:31:27.936037 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:31:27.935994 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:31:29.153672 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:29.153639 5091 scope.go:122] "RemoveContainer" containerID="a8fdcd005a1d61561fae0959888579f923eed75509e21c4f9f61d93c5e6bad8a" Nov 02 15:31:29.164652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:29.164630 5091 scope.go:122] "RemoveContainer" containerID="0cd1019b2843cad714cbb855d8ed7bc382008ff5a485ecbe5ceeb6199b3d749e" Nov 02 15:31:29.173755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:29.173722 5091 scope.go:122] "RemoveContainer" containerID="ef5a343ed96e5bae94e64dbbb64914363bd3d7603b23a7d30b410400acf3229a" Nov 02 15:31:29.182820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:29.182797 5091 scope.go:122] "RemoveContainer" containerID="9d72811a9b7270d42e7b9b833d497468d2da008e26c52db3b119b06e90ef8d03" Nov 02 15:31:30.294017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.293985 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp"] Nov 02 15:31:30.294656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294640 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294657 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294670 5091 cpu_manager.go:414] "containerMap: removing container" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294675 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294690 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294695 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294709 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.294714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294713 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294721 5091 cpu_manager.go:414] "containerMap: removing container" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294726 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294734 5091 cpu_manager.go:414] "containerMap: removing container" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294739 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294747 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294751 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294763 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294767 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294775 5091 cpu_manager.go:414] "containerMap: removing container" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294779 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294787 5091 cpu_manager.go:414] "containerMap: removing container" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294793 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294802 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294809 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294824 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294829 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294841 5091 cpu_manager.go:414] "containerMap: removing container" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294846 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294851 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294855 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294861 5091 cpu_manager.go:414] "containerMap: removing container" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:30.294926 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294865 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294972 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294979 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294986 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="514400cc-4295-410f-917c-3437fc726399" containerName="istio-proxy" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.294993 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295000 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295006 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295011 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295017 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295024 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295029 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295034 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295040 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295044 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:30.295674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.295051 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:30.300385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.300360 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.354670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.354643 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-adf096-openshift-default-dockercfg-kppng\"" Nov 02 15:31:30.400627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.400594 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp"] Nov 02 15:31:30.432509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432467 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432588 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432661 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432743 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432849 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432821 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432858 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsln\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432900 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.432956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.432943 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.534421 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.534386 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.534815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.534784 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.534972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.534955 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsln\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.535129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.535107 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.535284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.535270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.535502 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.535436 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.535856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.535823 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.536053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.536036 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.536195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.536181 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.536330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.536302 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.536559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.536075 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.536652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.536316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.537242 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.537216 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.537353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.537295 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.547177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.547096 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.549058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.549029 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.615540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.611105 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsln\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.615540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.612665 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token\") pod \"e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:30.641305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:30.640019 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:31.232523 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:31.232475 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp"] Nov 02 15:31:31.258585 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:31:31.258546 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88f8840_f89d_4ec4_ae7c_a688205a452e.slice/crio-e4aa112cafc08ad001450631ca086316ba24b9d00b6711694d5d80398a7ff67a WatchSource:0}: Error finding container e4aa112cafc08ad001450631ca086316ba24b9d00b6711694d5d80398a7ff67a: Status 404 returned error can't find the container with id e4aa112cafc08ad001450631ca086316ba24b9d00b6711694d5d80398a7ff67a Nov 02 15:31:31.261456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:31.261420 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:31.261586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:31.261491 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:31.261586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:31.261538 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:31:32.079168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:32.078629 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" event={"ID":"e88f8840-f89d-4ec4-ae7c-a688205a452e","Type":"ContainerStarted","Data":"ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc"} Nov 02 15:31:32.079168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:32.078712 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" event={"ID":"e88f8840-f89d-4ec4-ae7c-a688205a452e","Type":"ContainerStarted","Data":"e4aa112cafc08ad001450631ca086316ba24b9d00b6711694d5d80398a7ff67a"} Nov 02 15:31:32.317438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:32.317376 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podStartSLOduration=2.317360968 podStartE2EDuration="2.317360968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:32.269674098 +0000 UTC m=+2046.434226639" watchObservedRunningTime="2026-11-02 15:31:32.317360968 +0000 UTC m=+2046.481913509" Nov 02 15:31:32.640700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:32.640660 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:32.646678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:32.646646 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:33.085594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:33.085486 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:33.086454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:33.086436 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:31:45.692877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.692841 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4"] Nov 02 15:31:45.693389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.693031 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" containerID="cri-o://6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad" gracePeriod=30 Nov 02 15:31:45.806203 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806172 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm"] Nov 02 15:31:45.806853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806839 5091 cpu_manager.go:414] "containerMap: removing container" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:45.806913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806854 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="338d94ff-8f6a-432e-8d20-5fc1d45bf837" containerName="cleanup" Nov 02 15:31:45.806913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806868 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:45.806913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806873 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:45.806913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806898 5091 cpu_manager.go:414] "containerMap: removing container" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:45.806913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806903 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806924 5091 cpu_manager.go:414] "containerMap: removing container" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.806930 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.807023 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="375810a9-ebb9-43e5-ab63-898a086490ba" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.807033 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e30f1cb-4d42-4c20-937b-a4022764d9f5" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.807042 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c65ab52-478d-4600-8421-a51ffacde722" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.807050 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c180a15-d346-463e-9eaa-fd9b200c023e" containerName="cleanup" Nov 02 15:31:45.807067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.807058 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="72c647a6-3311-48f4-83e1-045b5e2f0061" containerName="cleanup" Nov 02 15:31:45.810938 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.810908 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.862255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.862218 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:31:45.862703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.862674 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:31:45.866928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.866902 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm"] Nov 02 15:31:45.896957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.896925 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.897074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.897053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv8t\" (UniqueName: \"kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.897213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.897191 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.998587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.998480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.998742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.998724 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.998784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.998772 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv8t\" (UniqueName: \"kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:45.999412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:45.999394 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:46.000928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:46.000903 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:46.057850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:46.057819 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv8t\" (UniqueName: \"kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t\") pod \"maas-api-revoke-sub-c96e6963-z29rm\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:46.127268 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:46.127237 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:46.512804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:46.512776 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm"] Nov 02 15:31:46.514802 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:31:46.514776 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185dc6e4_d74e_44c1_a5df_101e897a00cb.slice/crio-9efaa49f93f6bc899865896e2a9232225c37c894e37ec8a0b8bdcf4484a143c6 WatchSource:0}: Error finding container 9efaa49f93f6bc899865896e2a9232225c37c894e37ec8a0b8bdcf4484a143c6: Status 404 returned error can't find the container with id 9efaa49f93f6bc899865896e2a9232225c37c894e37ec8a0b8bdcf4484a143c6 Nov 02 15:31:47.232127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:47.232084 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" event={"ID":"185dc6e4-d74e-44c1-a5df-101e897a00cb","Type":"ContainerStarted","Data":"28072a4382cbba37459718831395ec2564dee6ed37fe35bdbb6693926f52f633"} Nov 02 15:31:47.232636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:47.232156 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" event={"ID":"185dc6e4-d74e-44c1-a5df-101e897a00cb","Type":"ContainerStarted","Data":"9efaa49f93f6bc899865896e2a9232225c37c894e37ec8a0b8bdcf4484a143c6"} Nov 02 15:31:47.359037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:47.358964 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" podStartSLOduration=2.358945695 podStartE2EDuration="2.358945695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:47.354174485 +0000 UTC m=+2061.518727039" watchObservedRunningTime="2026-11-02 15:31:47.358945695 +0000 UTC m=+2061.523498239" Nov 02 15:31:48.241830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:48.241797 5091 generic.go:356] "Generic (PLEG): container finished" podID="185dc6e4-d74e-44c1-a5df-101e897a00cb" containerID="28072a4382cbba37459718831395ec2564dee6ed37fe35bdbb6693926f52f633" exitCode=0 Nov 02 15:31:48.242261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:48.241874 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" event={"ID":"185dc6e4-d74e-44c1-a5df-101e897a00cb","Type":"ContainerDied","Data":"28072a4382cbba37459718831395ec2564dee6ed37fe35bdbb6693926f52f633"} Nov 02 15:31:49.078234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.078192 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:31:49.078403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.078250 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:31:49.247850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.247816 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf"] Nov 02 15:31:49.248407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.248098 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" containerID="cri-o://c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3" gracePeriod=30 Nov 02 15:31:49.434822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.434793 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:49.553540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.553477 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t\" (UniqueName: \"kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t\") pod \"185dc6e4-d74e-44c1-a5df-101e897a00cb\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " Nov 02 15:31:49.554132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.553849 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca\") pod \"185dc6e4-d74e-44c1-a5df-101e897a00cb\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " Nov 02 15:31:49.554132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.553979 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca\") pod \"185dc6e4-d74e-44c1-a5df-101e897a00cb\" (UID: \"185dc6e4-d74e-44c1-a5df-101e897a00cb\") " Nov 02 15:31:49.555270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.555245 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca" pod "185dc6e4-d74e-44c1-a5df-101e897a00cb" (UID: "185dc6e4-d74e-44c1-a5df-101e897a00cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:49.561115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.561033 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca" pod "185dc6e4-d74e-44c1-a5df-101e897a00cb" (UID: "185dc6e4-d74e-44c1-a5df-101e897a00cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:31:49.567009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.566973 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t" pod "185dc6e4-d74e-44c1-a5df-101e897a00cb" (UID: "185dc6e4-d74e-44c1-a5df-101e897a00cb"). InnerVolumeSpecName "kube-api-access-cfv8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:49.656780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.656707 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:49.656780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.656743 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/185dc6e4-d74e-44c1-a5df-101e897a00cb-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:49.656780 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:49.656760 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfv8t\" (UniqueName: \"kubernetes.io/projected/185dc6e4-d74e-44c1-a5df-101e897a00cb-kube-api-access-cfv8t\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:50.264366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:50.264291 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" Nov 02 15:31:50.264366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:50.264319 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm" event={"ID":"185dc6e4-d74e-44c1-a5df-101e897a00cb","Type":"ContainerDied","Data":"9efaa49f93f6bc899865896e2a9232225c37c894e37ec8a0b8bdcf4484a143c6"} Nov 02 15:31:50.264795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:50.264369 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9efaa49f93f6bc899865896e2a9232225c37c894e37ec8a0b8bdcf4484a143c6" Nov 02 15:31:50.977505 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:50.977442 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:31:51.078297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078260 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078371 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078411 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078439 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.078736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078680 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078751 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.078794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078779 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078850 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.078908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078850 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:51.078908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078888 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token\") pod \"9c4479f9-c787-48df-beb8-2f9283329572\" (UID: \"9c4479f9-c787-48df-beb8-2f9283329572\") " Nov 02 15:31:51.079119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.078977 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.079220 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.079196 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.079640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.079614 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.079752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.079644 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.079752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.079660 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c4479f9-c787-48df-beb8-2f9283329572-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.079752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.079674 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.079752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.079690 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.080981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.080955 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "kube-api-access-vwfmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:51.080981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.080965 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:51.081234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.081210 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:51.081336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.081265 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token" pod "9c4479f9-c787-48df-beb8-2f9283329572" (UID: "9c4479f9-c787-48df-beb8-2f9283329572"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:51.181595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.181556 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c4479f9-c787-48df-beb8-2f9283329572-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.181595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.181596 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c4479f9-c787-48df-beb8-2f9283329572-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.181830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.181611 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.181830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.181625 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwfmg\" (UniqueName: \"kubernetes.io/projected/9c4479f9-c787-48df-beb8-2f9283329572-kube-api-access-vwfmg\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:51.278950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.276034 5091 generic.go:356] "Generic (PLEG): container finished" podID="9c4479f9-c787-48df-beb8-2f9283329572" containerID="6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad" exitCode=0 Nov 02 15:31:51.279481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.279448 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" event={"ID":"9c4479f9-c787-48df-beb8-2f9283329572","Type":"ContainerDied","Data":"6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad"} Nov 02 15:31:51.279632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.279542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" event={"ID":"9c4479f9-c787-48df-beb8-2f9283329572","Type":"ContainerDied","Data":"a78ee55380b1e38cfa874b8183c65d09beda02bb8490036a2370508fd8df73e7"} Nov 02 15:31:51.279632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.279579 5091 scope.go:122] "RemoveContainer" containerID="6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad" Nov 02 15:31:51.279782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.279760 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4" Nov 02 15:31:51.292470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.292450 5091 scope.go:122] "RemoveContainer" containerID="6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad" Nov 02 15:31:51.292767 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:31:51.292745 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad\": container with ID starting with 6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad not found: ID does not exist" containerID="6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad" Nov 02 15:31:51.292838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.292774 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad"} err="failed to get container status \"6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad\": rpc error: code = NotFound desc = could not find container \"6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad\": container with ID starting with 6c669606a3f0e36cede16b304f211dc4ea131bfbcaa87a82f86d17a70c7c3fad not found: ID does not exist" Nov 02 15:31:51.454479 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.454442 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4"] Nov 02 15:31:51.458639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.458613 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-z4km4"] Nov 02 15:31:51.965894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:51.965860 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4479f9-c787-48df-beb8-2f9283329572" path="/var/lib/kubelet/pods/9c4479f9-c787-48df-beb8-2f9283329572/volumes" Nov 02 15:31:54.540946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.540885 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:31:54.625163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625130 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625197 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625298 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625340 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625407 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625420 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.625548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625487 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625572 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.625696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625583 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625652 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.625911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625742 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo\") pod \"24b580d5-d425-4fca-8453-7ea6e61e2a75\" (UID: \"24b580d5-d425-4fca-8453-7ea6e61e2a75\") " Nov 02 15:31:54.625911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.625849 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.626111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.626078 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:31:54.626415 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.626390 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.626557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.626421 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.626557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.626436 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.626557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.626449 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24b580d5-d425-4fca-8453-7ea6e61e2a75-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.626557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.626463 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.628005 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.627982 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "kube-api-access-4kl4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:54.628231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.628207 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:31:54.628461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.628441 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:31:54.628546 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.628508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy" pod "24b580d5-d425-4fca-8453-7ea6e61e2a75" (UID: "24b580d5-d425-4fca-8453-7ea6e61e2a75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:31:54.728110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.728069 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.728110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.728101 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.728110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.728112 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24b580d5-d425-4fca-8453-7ea6e61e2a75-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:54.728416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:54.728121 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kl4w\" (UniqueName: \"kubernetes.io/projected/24b580d5-d425-4fca-8453-7ea6e61e2a75-kube-api-access-4kl4w\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:31:55.134107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134071 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6582-b983a5"] Nov 02 15:31:55.134763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134748 5091 cpu_manager.go:414] "containerMap: removing container" podUID="185dc6e4-d74e-44c1-a5df-101e897a00cb" containerName="revoke-keys" Nov 02 15:31:55.134822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134764 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185dc6e4-d74e-44c1-a5df-101e897a00cb" containerName="revoke-keys" Nov 02 15:31:55.134822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134783 5091 cpu_manager.go:414] "containerMap: removing container" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" Nov 02 15:31:55.134822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134788 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" Nov 02 15:31:55.134822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134816 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" Nov 02 15:31:55.134822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134820 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" Nov 02 15:31:55.135039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134925 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="185dc6e4-d74e-44c1-a5df-101e897a00cb" containerName="revoke-keys" Nov 02 15:31:55.135039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134942 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerName="istio-proxy" Nov 02 15:31:55.135039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.134954 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4479f9-c787-48df-beb8-2f9283329572" containerName="istio-proxy" Nov 02 15:31:55.139013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.138993 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:55.188904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.188873 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xhrbh\"" Nov 02 15:31:55.190679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.190651 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6582-b983a5"] Nov 02 15:31:55.236535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.236483 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rln\" (UniqueName: \"kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln\") pod \"test-curl-6582-b983a5\" (UID: \"c80c5226-ee40-441b-a237-abd6a7e932fb\") " pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:55.309000 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.308969 5091 generic.go:356] "Generic (PLEG): container finished" podID="24b580d5-d425-4fca-8453-7ea6e61e2a75" containerID="c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3" exitCode=0 Nov 02 15:31:55.309168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.309012 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" Nov 02 15:31:55.309168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.309053 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" event={"ID":"24b580d5-d425-4fca-8453-7ea6e61e2a75","Type":"ContainerDied","Data":"c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3"} Nov 02 15:31:55.309168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.309110 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf" event={"ID":"24b580d5-d425-4fca-8453-7ea6e61e2a75","Type":"ContainerDied","Data":"2e268c8b9c9017b35f14c9e22834bb8ed3456351b0cf3e51ad314d002083a95d"} Nov 02 15:31:55.309168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.309139 5091 scope.go:122] "RemoveContainer" containerID="c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3" Nov 02 15:31:55.319699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.319683 5091 scope.go:122] "RemoveContainer" containerID="c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3" Nov 02 15:31:55.319939 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:31:55.319922 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3\": container with ID starting with c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3 not found: ID does not exist" containerID="c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3" Nov 02 15:31:55.319992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.319947 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3"} err="failed to get container status \"c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3\": rpc error: code = NotFound desc = could not find container \"c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3\": container with ID starting with c46d697f6eef56bf107829080ba61d278d74ade36873aae78542d2fd8c7d80d3 not found: ID does not exist" Nov 02 15:31:55.338062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.338033 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rln\" (UniqueName: \"kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln\") pod \"test-curl-6582-b983a5\" (UID: \"c80c5226-ee40-441b-a237-abd6a7e932fb\") " pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:55.396713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.396648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rln\" (UniqueName: \"kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln\") pod \"test-curl-6582-b983a5\" (UID: \"c80c5226-ee40-441b-a237-abd6a7e932fb\") " pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:55.452110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.452069 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:55.478558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.478370 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf"] Nov 02 15:31:55.483482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.483453 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-8fcnf"] Nov 02 15:31:55.838676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.838595 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6582-b983a5"] Nov 02 15:31:55.843819 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:31:55.843787 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc80c5226_ee40_441b_a237_abd6a7e932fb.slice/crio-ed1f8713a10acbc23545992aecbac6ad3fce62e07183ba4e52c14d5a53c0edc1 WatchSource:0}: Error finding container ed1f8713a10acbc23545992aecbac6ad3fce62e07183ba4e52c14d5a53c0edc1: Status 404 returned error can't find the container with id ed1f8713a10acbc23545992aecbac6ad3fce62e07183ba4e52c14d5a53c0edc1 Nov 02 15:31:55.965288 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:55.965250 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b580d5-d425-4fca-8453-7ea6e61e2a75" path="/var/lib/kubelet/pods/24b580d5-d425-4fca-8453-7ea6e61e2a75/volumes" Nov 02 15:31:56.318815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:56.318778 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6582-b983a5" event={"ID":"c80c5226-ee40-441b-a237-abd6a7e932fb","Type":"ContainerStarted","Data":"03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be"} Nov 02 15:31:56.318815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:56.318820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6582-b983a5" event={"ID":"c80c5226-ee40-441b-a237-abd6a7e932fb","Type":"ContainerStarted","Data":"ed1f8713a10acbc23545992aecbac6ad3fce62e07183ba4e52c14d5a53c0edc1"} Nov 02 15:31:56.432689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:56.432632 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6582-b983a5" podStartSLOduration=1.432615612 podStartE2EDuration="1.432615612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:31:56.427453495 +0000 UTC m=+2070.592006042" watchObservedRunningTime="2026-11-02 15:31:56.432615612 +0000 UTC m=+2070.597168153" Nov 02 15:31:57.939126 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:31:57.939075 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:31:58.000431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.000392 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6582-b983a5"] Nov 02 15:31:58.029702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.029669 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6582-b983a5"] Nov 02 15:31:58.339886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.339780 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6582-b983a5" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" containerName="test-curl-6582-b983a5" containerID="cri-o://03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be" gracePeriod=2 Nov 02 15:31:58.388764 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:31:58.388713 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6582-b983a5\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:58.931802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.931767 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl"] Nov 02 15:31:58.932442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.932422 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" containerName="test-curl-6582-b983a5" Nov 02 15:31:58.932603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.932444 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" containerName="test-curl-6582-b983a5" Nov 02 15:31:58.932707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.932692 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" containerName="test-curl-6582-b983a5" Nov 02 15:31:58.936509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:58.936488 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.035438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.035390 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:31:59.035939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.035493 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:31:59.041624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.041593 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl"] Nov 02 15:31:59.096239 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:31:59.096206 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6582-b983a5\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:31:59.111512 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.111487 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgck\" (UniqueName: \"kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.111644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.111618 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.111752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.111737 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.212894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.212814 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgck\" (UniqueName: \"kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.212894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.212880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.213127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.212952 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.213601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.213577 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.215398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.215363 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.271583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.271550 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgck\" (UniqueName: \"kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.554344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.554267 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:31:59.929133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:31:59.929093 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl"] Nov 02 15:31:59.933570 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:31:59.933505 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode300c2af_2b85_4e4d_b982_78b5ca6196ea.slice/crio-8e68f2d4a41ce21fbac89ae5c78152354ef00a074578e711eb7f1b123dd4100c WatchSource:0}: Error finding container 8e68f2d4a41ce21fbac89ae5c78152354ef00a074578e711eb7f1b123dd4100c: Status 404 returned error can't find the container with id 8e68f2d4a41ce21fbac89ae5c78152354ef00a074578e711eb7f1b123dd4100c Nov 02 15:32:00.357227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.357187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" event={"ID":"e300c2af-2b85-4e4d-b982-78b5ca6196ea","Type":"ContainerStarted","Data":"cea0652e0eeb1a90adc8aeb1d8c96053e306abbea4cb99b82bd7bb0d220ed5db"} Nov 02 15:32:00.357633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.357284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" event={"ID":"e300c2af-2b85-4e4d-b982-78b5ca6196ea","Type":"ContainerStarted","Data":"8e68f2d4a41ce21fbac89ae5c78152354ef00a074578e711eb7f1b123dd4100c"} Nov 02 15:32:00.471565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.471496 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" podStartSLOduration=2.471483303 podStartE2EDuration="2.471483303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:00.468159241 +0000 UTC m=+2074.632711784" watchObservedRunningTime="2026-11-02 15:32:00.471483303 +0000 UTC m=+2074.636035843" Nov 02 15:32:00.606568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.606537 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:32:00.636957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.636920 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln\" (UniqueName: \"kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln\") pod \"c80c5226-ee40-441b-a237-abd6a7e932fb\" (UID: \"c80c5226-ee40-441b-a237-abd6a7e932fb\") " Nov 02 15:32:00.638998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.638968 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln" pod "c80c5226-ee40-441b-a237-abd6a7e932fb" (UID: "c80c5226-ee40-441b-a237-abd6a7e932fb"). InnerVolumeSpecName "kube-api-access-f9rln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:00.738815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.738785 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rln\" (UniqueName: \"kubernetes.io/projected/c80c5226-ee40-441b-a237-abd6a7e932fb-kube-api-access-f9rln\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:00.813842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.813772 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v"] Nov 02 15:32:00.813984 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:00.813961 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" containerID="cri-o://f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e" gracePeriod=30 Nov 02 15:32:01.367943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.367910 5091 generic.go:356] "Generic (PLEG): container finished" podID="e300c2af-2b85-4e4d-b982-78b5ca6196ea" containerID="cea0652e0eeb1a90adc8aeb1d8c96053e306abbea4cb99b82bd7bb0d220ed5db" exitCode=0 Nov 02 15:32:01.368370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.367997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" event={"ID":"e300c2af-2b85-4e4d-b982-78b5ca6196ea","Type":"ContainerDied","Data":"cea0652e0eeb1a90adc8aeb1d8c96053e306abbea4cb99b82bd7bb0d220ed5db"} Nov 02 15:32:01.369655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.369632 5091 generic.go:356] "Generic (PLEG): container finished" podID="c80c5226-ee40-441b-a237-abd6a7e932fb" containerID="03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be" exitCode=137 Nov 02 15:32:01.369787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.369678 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:32:01.369787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.369738 5091 scope.go:122] "RemoveContainer" containerID="03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be" Nov 02 15:32:01.384330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.383746 5091 scope.go:122] "RemoveContainer" containerID="03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be" Nov 02 15:32:01.384330 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:01.384145 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be\": container with ID starting with 03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be not found: ID does not exist" containerID="03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be" Nov 02 15:32:01.384330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.384176 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be"} err="failed to get container status \"03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be\": rpc error: code = NotFound desc = could not find container \"03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be\": container with ID starting with 03113ad7c427a8799c38eebae139c817a0a136f21a1b1c026796ae2e8e3607be not found: ID does not exist" Nov 02 15:32:01.549698 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:01.549657 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6582-b983a5\" is forbidden: User \"system:node:ip-10-0-130-229.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-229.us-west-2.compute.internal' and this object" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" pod="opendatahub/test-curl-6582-b983a5" Nov 02 15:32:01.961305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:01.961265 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80c5226-ee40-441b-a237-abd6a7e932fb" path="/var/lib/kubelet/pods/c80c5226-ee40-441b-a237-abd6a7e932fb/volumes" Nov 02 15:32:02.542502 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.542477 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:32:02.564560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.564534 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca\") pod \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " Nov 02 15:32:02.564713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.564599 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca\") pod \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " Nov 02 15:32:02.564779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.564740 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck\" (UniqueName: \"kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck\") pod \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\" (UID: \"e300c2af-2b85-4e4d-b982-78b5ca6196ea\") " Nov 02 15:32:02.564986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.564962 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca" pod "e300c2af-2b85-4e4d-b982-78b5ca6196ea" (UID: "e300c2af-2b85-4e4d-b982-78b5ca6196ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:02.565278 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.565261 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.566653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.566632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca" pod "e300c2af-2b85-4e4d-b982-78b5ca6196ea" (UID: "e300c2af-2b85-4e4d-b982-78b5ca6196ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:32:02.566813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.566796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck" pod "e300c2af-2b85-4e4d-b982-78b5ca6196ea" (UID: "e300c2af-2b85-4e4d-b982-78b5ca6196ea"). InnerVolumeSpecName "kube-api-access-7qgck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:02.666212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.666136 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgck\" (UniqueName: \"kubernetes.io/projected/e300c2af-2b85-4e4d-b982-78b5ca6196ea-kube-api-access-7qgck\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:02.666212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:02.666166 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e300c2af-2b85-4e4d-b982-78b5ca6196ea-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:03.388705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:03.388666 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" Nov 02 15:32:03.389271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:03.388673 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl" event={"ID":"e300c2af-2b85-4e4d-b982-78b5ca6196ea","Type":"ContainerDied","Data":"8e68f2d4a41ce21fbac89ae5c78152354ef00a074578e711eb7f1b123dd4100c"} Nov 02 15:32:03.389373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:03.389304 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e68f2d4a41ce21fbac89ae5c78152354ef00a074578e711eb7f1b123dd4100c" Nov 02 15:32:04.351732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:04.351687 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht"] Nov 02 15:32:04.352677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:04.352638 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" containerID="cri-o://3484f2b83efe0a69349173ec721a6cf044d52df99afa5dd25b1168af1d4c015f" gracePeriod=30 Nov 02 15:32:05.695051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:05.695007 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:32:05.696085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:05.695622 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:32:06.106188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.106158 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:32:06.198446 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198415 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.198765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.198765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198492 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.198765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198558 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.198765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198728 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.198765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198736 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.199066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198771 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.199066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.198884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.199174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199066 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.199174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199085 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:06.199174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199120 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.199340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199181 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket\") pod \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\" (UID: \"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a\") " Nov 02 15:32:06.199695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199668 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.199927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199871 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.199927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199825 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.200093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199949 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.200093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.199965 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.201088 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.201064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:06.201177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.201083 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:06.201406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.201375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:32:06.201450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.201402 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn" pod "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" (UID: "0a4c0a84-d2e4-47cd-a333-1d1cc10b517a"). InnerVolumeSpecName "kube-api-access-p6wqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:06.301547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.301432 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.301547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.301470 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6wqn\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-kube-api-access-p6wqn\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.301547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.301487 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.301547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.301501 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.301838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.301553 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.301838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.301567 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:06.524339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.524303 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerID="f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e" exitCode=0 Nov 02 15:32:06.524545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.524351 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" Nov 02 15:32:06.524545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.524391 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" event={"ID":"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a","Type":"ContainerDied","Data":"f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e"} Nov 02 15:32:06.524545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.524451 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v" event={"ID":"0a4c0a84-d2e4-47cd-a333-1d1cc10b517a","Type":"ContainerDied","Data":"7459c0683526fe81ac0429bdd79707d7d053efc9455c4e7a0bad4d8eba8dd653"} Nov 02 15:32:06.524545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.524478 5091 scope.go:122] "RemoveContainer" containerID="f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e" Nov 02 15:32:06.536348 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.536326 5091 scope.go:122] "RemoveContainer" containerID="f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e" Nov 02 15:32:06.536743 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:06.536718 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e\": container with ID starting with f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e not found: ID does not exist" containerID="f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e" Nov 02 15:32:06.536825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.536757 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e"} err="failed to get container status \"f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e\": rpc error: code = NotFound desc = could not find container \"f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e\": container with ID starting with f3645664d03c50a192e2e28ebafe2cbb7c412a9eb01b5c312286e3f7a15c7f9e not found: ID does not exist" Nov 02 15:32:06.691394 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.691356 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v"] Nov 02 15:32:06.699564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:06.699500 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-7ng4v"] Nov 02 15:32:07.964340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:07.964305 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" path="/var/lib/kubelet/pods/0a4c0a84-d2e4-47cd-a333-1d1cc10b517a/volumes" Nov 02 15:32:09.548459 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.548416 5091 generic.go:356] "Generic (PLEG): container finished" podID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerID="3484f2b83efe0a69349173ec721a6cf044d52df99afa5dd25b1168af1d4c015f" exitCode=0 Nov 02 15:32:09.548854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.548483 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" event={"ID":"33b1e9b3-1988-458f-af39-7c5d0be754b3","Type":"ContainerDied","Data":"3484f2b83efe0a69349173ec721a6cf044d52df99afa5dd25b1168af1d4c015f"} Nov 02 15:32:09.611782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.611761 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:32:09.757139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757068 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757134 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757171 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757275 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757327 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757380 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757434 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757503 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757568 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:09.757827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757614 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:09.757827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757663 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo\") pod \"33b1e9b3-1988-458f-af39-7c5d0be754b3\" (UID: \"33b1e9b3-1988-458f-af39-7c5d0be754b3\") " Nov 02 15:32:09.757827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757683 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:09.757827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:09.758048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.757950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:09.758314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.758286 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.758429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.758318 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.758429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.758337 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.758429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.758352 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33b1e9b3-1988-458f-af39-7c5d0be754b3-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.758429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.758365 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.759548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.759486 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "kube-api-access-swqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:09.759681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.759655 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:09.759824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.759804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:32:09.759969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.759953 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy" pod "33b1e9b3-1988-458f-af39-7c5d0be754b3" (UID: "33b1e9b3-1988-458f-af39-7c5d0be754b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:32:09.860148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.860115 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swqf7\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-kube-api-access-swqf7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.860148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.860144 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.860148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.860154 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:09.860383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:09.860164 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33b1e9b3-1988-458f-af39-7c5d0be754b3-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:10.557124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:10.557096 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" Nov 02 15:32:10.557124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:10.557102 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht" event={"ID":"33b1e9b3-1988-458f-af39-7c5d0be754b3","Type":"ContainerDied","Data":"15df99a28773bd489110f1efe6bc1c54c029f9bc2b61a558a98e42af2c9e0fd3"} Nov 02 15:32:10.557661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:10.557169 5091 scope.go:122] "RemoveContainer" containerID="3484f2b83efe0a69349173ec721a6cf044d52df99afa5dd25b1168af1d4c015f" Nov 02 15:32:10.813907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:10.813826 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht"] Nov 02 15:32:10.835137 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:10.835097 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-qqnht"] Nov 02 15:32:10.862603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:10.862581 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d"] Nov 02 15:32:11.936357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936324 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn"] Nov 02 15:32:11.936932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936917 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e300c2af-2b85-4e4d-b982-78b5ca6196ea" containerName="revoke-keys" Nov 02 15:32:11.936932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936931 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e300c2af-2b85-4e4d-b982-78b5ca6196ea" containerName="revoke-keys" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936950 5091 cpu_manager.go:414] "containerMap: removing container" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936955 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936968 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.936973 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.937079 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4c0a84-d2e4-47cd-a333-1d1cc10b517a" containerName="istio-proxy" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.937092 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e300c2af-2b85-4e4d-b982-78b5ca6196ea" containerName="revoke-keys" Nov 02 15:32:11.937122 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.937100 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" containerName="istio-proxy" Nov 02 15:32:11.941858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.941834 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:11.962858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.962829 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33b1e9b3-1988-458f-af39-7c5d0be754b3" path="/var/lib/kubelet/pods/33b1e9b3-1988-458f-af39-7c5d0be754b3/volumes" Nov 02 15:32:11.991838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.991814 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:32:11.992311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.992289 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:32:11.995388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:11.995362 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn"] Nov 02 15:32:12.090756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.090721 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.090906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.090890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzsx\" (UniqueName: \"kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.090967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.090935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.192864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.192780 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.193013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.192880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.193054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.193010 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzsx\" (UniqueName: \"kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.193428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.193408 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.195464 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.195440 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.270584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.270552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzsx\" (UniqueName: \"kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx\") pod \"maas-api-revoke-sub-cdf9737e-rrlqn\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.396861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.396825 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq"] Nov 02 15:32:12.402373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.402348 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.453614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.453506 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-fb80ca-serving-cert\"" Nov 02 15:32:12.465431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.465374 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq"] Nov 02 15:32:12.501689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.501628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.501999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.501973 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.502198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.502179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.502429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.502414 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4tg\" (UniqueName: \"kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.559774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.559733 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:12.604281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.604241 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4tg\" (UniqueName: \"kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.604432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.604400 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.604508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.604457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.604634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.604549 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.604855 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:12.604792 5091 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-serving-cert: secret "maas-api-e2e-shared-b-w6-fb80ca-serving-cert" not found Nov 02 15:32:12.604980 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:12.604873 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls podName:56a78bf5-2886-4938-b220-f4775b170a36 nodeName:}" failed. No retries permitted until 2026-11-02 15:32:13.104852581 +0000 UTC m=+2087.269405116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls") pod "maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" (UID: "56a78bf5-2886-4938-b220-f4775b170a36") : secret "maas-api-e2e-shared-b-w6-fb80ca-serving-cert" not found Nov 02 15:32:12.605469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.605440 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.608085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.608051 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.684635 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.684560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4tg\" (UniqueName: \"kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:12.970307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:12.970243 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn"] Nov 02 15:32:12.974819 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:32:12.974793 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764b0102_b952_40b0_998a_d918ea06512d.slice/crio-7f24e17ef45d3dcd2fd01415dcc60ab779ddc81ac74c2601499a913131138087 WatchSource:0}: Error finding container 7f24e17ef45d3dcd2fd01415dcc60ab779ddc81ac74c2601499a913131138087: Status 404 returned error can't find the container with id 7f24e17ef45d3dcd2fd01415dcc60ab779ddc81ac74c2601499a913131138087 Nov 02 15:32:13.115560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.115501 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:13.118194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.118166 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:13.322083 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.322052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:13.591877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.591794 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" event={"ID":"764b0102-b952-40b0-998a-d918ea06512d","Type":"ContainerStarted","Data":"e04e381279841b761111dcfc9a896f859c44eb0dbaee007ddf40abfe6539ba8c"} Nov 02 15:32:13.591877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.591858 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" event={"ID":"764b0102-b952-40b0-998a-d918ea06512d","Type":"ContainerStarted","Data":"7f24e17ef45d3dcd2fd01415dcc60ab779ddc81ac74c2601499a913131138087"} Nov 02 15:32:13.703813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.703777 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq"] Nov 02 15:32:13.707737 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:32:13.707702 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a78bf5_2886_4938_b220_f4775b170a36.slice/crio-8c00829025ac48d3e42be825d61e17b44a54ec6912e268a1a3023743befd2807 WatchSource:0}: Error finding container 8c00829025ac48d3e42be825d61e17b44a54ec6912e268a1a3023743befd2807: Status 404 returned error can't find the container with id 8c00829025ac48d3e42be825d61e17b44a54ec6912e268a1a3023743befd2807 Nov 02 15:32:13.709560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.709526 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:32:13.749561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:13.749487 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" podStartSLOduration=2.749474216 podStartE2EDuration="2.749474216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:13.702117627 +0000 UTC m=+2087.866670167" watchObservedRunningTime="2026-11-02 15:32:13.749474216 +0000 UTC m=+2087.914026756" Nov 02 15:32:14.599689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:14.599636 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" event={"ID":"56a78bf5-2886-4938-b220-f4775b170a36","Type":"ContainerStarted","Data":"8c00829025ac48d3e42be825d61e17b44a54ec6912e268a1a3023743befd2807"} Nov 02 15:32:14.601540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:14.601496 5091 generic.go:356] "Generic (PLEG): container finished" podID="764b0102-b952-40b0-998a-d918ea06512d" containerID="e04e381279841b761111dcfc9a896f859c44eb0dbaee007ddf40abfe6539ba8c" exitCode=0 Nov 02 15:32:14.601700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:14.601561 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" event={"ID":"764b0102-b952-40b0-998a-d918ea06512d","Type":"ContainerDied","Data":"e04e381279841b761111dcfc9a896f859c44eb0dbaee007ddf40abfe6539ba8c"} Nov 02 15:32:15.611728 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.611691 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" event={"ID":"56a78bf5-2886-4938-b220-f4775b170a36","Type":"ContainerStarted","Data":"7b903a95e55bb714445bd3bfa805da5ebc49fe1926dd3cc0de6f544cf2d1c875"} Nov 02 15:32:15.612230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.611783 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:15.812666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.802781 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" podStartSLOduration=2.881424608 podStartE2EDuration="3.802756189s" totalImagesPullingTime="921.331581ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:32:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:32:15.738815567 +0000 UTC m=+2089.903368115" watchObservedRunningTime="2026-11-02 15:32:15.802756189 +0000 UTC m=+2089.967308731" Nov 02 15:32:15.856617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.855433 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:15.871561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.870620 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerName="payload-processing" containerID="cri-o://e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b" gracePeriod=25 Nov 02 15:32:15.957414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.957380 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca\") pod \"764b0102-b952-40b0-998a-d918ea06512d\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " Nov 02 15:32:15.957634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.957458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx\" (UniqueName: \"kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx\") pod \"764b0102-b952-40b0-998a-d918ea06512d\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " Nov 02 15:32:15.957634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.957508 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca\") pod \"764b0102-b952-40b0-998a-d918ea06512d\" (UID: \"764b0102-b952-40b0-998a-d918ea06512d\") " Nov 02 15:32:15.958193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.958154 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca" pod "764b0102-b952-40b0-998a-d918ea06512d" (UID: "764b0102-b952-40b0-998a-d918ea06512d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:15.960152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.960127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca" pod "764b0102-b952-40b0-998a-d918ea06512d" (UID: "764b0102-b952-40b0-998a-d918ea06512d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:32:15.960811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:15.960784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx" pod "764b0102-b952-40b0-998a-d918ea06512d" (UID: "764b0102-b952-40b0-998a-d918ea06512d"). InnerVolumeSpecName "kube-api-access-gvzsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:16.059723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:16.059694 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvzsx\" (UniqueName: \"kubernetes.io/projected/764b0102-b952-40b0-998a-d918ea06512d-kube-api-access-gvzsx\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:16.059723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:16.059719 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/764b0102-b952-40b0-998a-d918ea06512d-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:16.059723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:16.059729 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/764b0102-b952-40b0-998a-d918ea06512d-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:16.620186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:16.620153 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" Nov 02 15:32:16.620712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:16.620189 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn" event={"ID":"764b0102-b952-40b0-998a-d918ea06512d","Type":"ContainerDied","Data":"7f24e17ef45d3dcd2fd01415dcc60ab779ddc81ac74c2601499a913131138087"} Nov 02 15:32:16.620712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:16.620259 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f24e17ef45d3dcd2fd01415dcc60ab779ddc81ac74c2601499a913131138087" Nov 02 15:32:21.626214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:21.626184 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:32:21.819859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:21.819817 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:32:27.937460 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:27.937419 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:32:28.631693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.631659 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz"] Nov 02 15:32:28.632500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.632476 5091 cpu_manager.go:414] "containerMap: removing container" podUID="764b0102-b952-40b0-998a-d918ea06512d" containerName="revoke-keys" Nov 02 15:32:28.632500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.632500 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="764b0102-b952-40b0-998a-d918ea06512d" containerName="revoke-keys" Nov 02 15:32:28.632750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.632669 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="764b0102-b952-40b0-998a-d918ea06512d" containerName="revoke-keys" Nov 02 15:32:28.635061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.635033 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:32:28.645953 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.645482 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.657657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.657627 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:32:28.698397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.698370 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz"] Nov 02 15:32:28.737558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.736130 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.737558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.736268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.737558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.736575 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thcx\" (UniqueName: \"kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.740493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.740464 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:32:28.740628 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.740502 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:32:28.837909 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.837873 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.838081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.838055 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thcx\" (UniqueName: \"kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.838171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.838152 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.838778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.838756 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.840327 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.840304 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.897426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.897359 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thcx\" (UniqueName: \"kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx\") pod \"maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:28.982068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:28.982032 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:29.381778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:29.381735 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz"] Nov 02 15:32:29.402022 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:32:29.401995 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0c7010_afcd_4ebe_abfb_9517ea0d0806.slice/crio-3dbc906fe066b9717d277c2cde54b3cd2ff0591039abafed7689408915452360 WatchSource:0}: Error finding container 3dbc906fe066b9717d277c2cde54b3cd2ff0591039abafed7689408915452360: Status 404 returned error can't find the container with id 3dbc906fe066b9717d277c2cde54b3cd2ff0591039abafed7689408915452360 Nov 02 15:32:29.738397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:29.738364 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" event={"ID":"8f0c7010-afcd-4ebe-abfb-9517ea0d0806","Type":"ContainerStarted","Data":"3dbc906fe066b9717d277c2cde54b3cd2ff0591039abafed7689408915452360"} Nov 02 15:32:30.751622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:30.751528 5091 generic.go:356] "Generic (PLEG): container finished" podID="8f0c7010-afcd-4ebe-abfb-9517ea0d0806" containerID="12cc24b7bed01368359b70af7194f6411ea2d6bc68690edd093dee2056f465c7" exitCode=0 Nov 02 15:32:30.751622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:30.751546 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" event={"ID":"8f0c7010-afcd-4ebe-abfb-9517ea0d0806","Type":"ContainerDied","Data":"12cc24b7bed01368359b70af7194f6411ea2d6bc68690edd093dee2056f465c7"} Nov 02 15:32:31.819894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:31.819600 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:32:32.225440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.225347 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:32.396367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.396271 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca\") pod \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " Nov 02 15:32:32.397426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.397402 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx\" (UniqueName: \"kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx\") pod \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " Nov 02 15:32:32.405432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.405396 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca\") pod \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\" (UID: \"8f0c7010-afcd-4ebe-abfb-9517ea0d0806\") " Nov 02 15:32:32.409160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.407952 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca" pod "8f0c7010-afcd-4ebe-abfb-9517ea0d0806" (UID: "8f0c7010-afcd-4ebe-abfb-9517ea0d0806"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:32.409806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.409786 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:32.419626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.418759 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca" pod "8f0c7010-afcd-4ebe-abfb-9517ea0d0806" (UID: "8f0c7010-afcd-4ebe-abfb-9517ea0d0806"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:32:32.419626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.419442 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx" pod "8f0c7010-afcd-4ebe-abfb-9517ea0d0806" (UID: "8f0c7010-afcd-4ebe-abfb-9517ea0d0806"). InnerVolumeSpecName "kube-api-access-2thcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:32.569540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.543665 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:32.569540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.543706 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thcx\" (UniqueName: \"kubernetes.io/projected/8f0c7010-afcd-4ebe-abfb-9517ea0d0806-kube-api-access-2thcx\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:32.795764 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.795545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" event={"ID":"8f0c7010-afcd-4ebe-abfb-9517ea0d0806","Type":"ContainerDied","Data":"3dbc906fe066b9717d277c2cde54b3cd2ff0591039abafed7689408915452360"} Nov 02 15:32:32.795764 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.795658 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dbc906fe066b9717d277c2cde54b3cd2ff0591039abafed7689408915452360" Nov 02 15:32:32.795993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:32.795964 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz" Nov 02 15:32:36.154034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.153973 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:32:36.190882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.190859 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume\") pod \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " Nov 02 15:32:36.191018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.190906 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r\" (UniqueName: \"kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r\") pod \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\" (UID: \"c0b4a363-0c44-454a-8c6d-fd5b059d4890\") " Nov 02 15:32:36.191237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.191212 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume" pod "c0b4a363-0c44-454a-8c6d-fd5b059d4890" (UID: "c0b4a363-0c44-454a-8c6d-fd5b059d4890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:36.191340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.191327 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b4a363-0c44-454a-8c6d-fd5b059d4890-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:36.192982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.192957 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r" pod "c0b4a363-0c44-454a-8c6d-fd5b059d4890" (UID: "c0b4a363-0c44-454a-8c6d-fd5b059d4890"). InnerVolumeSpecName "kube-api-access-5pb5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:36.292663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.292620 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pb5r\" (UniqueName: \"kubernetes.io/projected/c0b4a363-0c44-454a-8c6d-fd5b059d4890-kube-api-access-5pb5r\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:36.825269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.825235 5091 generic.go:356] "Generic (PLEG): container finished" podID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerID="e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b" exitCode=0 Nov 02 15:32:36.825447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.825278 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" Nov 02 15:32:36.825447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.825311 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" event={"ID":"c0b4a363-0c44-454a-8c6d-fd5b059d4890","Type":"ContainerDied","Data":"e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b"} Nov 02 15:32:36.825447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.825363 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d" event={"ID":"c0b4a363-0c44-454a-8c6d-fd5b059d4890","Type":"ContainerDied","Data":"e7009ce0d13f46ca44bbaeb6dc58e6bf84fc2dac1158402bb3fd0dd826d0a7e2"} Nov 02 15:32:36.825447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.825389 5091 scope.go:122] "RemoveContainer" containerID="e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b" Nov 02 15:32:36.836673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.836636 5091 scope.go:122] "RemoveContainer" containerID="e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b" Nov 02 15:32:36.836958 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:36.836928 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b\": container with ID starting with e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b not found: ID does not exist" containerID="e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b" Nov 02 15:32:36.837048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.836969 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b"} err="failed to get container status \"e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b\": rpc error: code = NotFound desc = could not find container \"e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b\": container with ID starting with e5746262c51a3ff27efc01292e4f99dcc6a2baae0db068d5fdd151540a378d5b not found: ID does not exist" Nov 02 15:32:36.983529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:36.983491 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d"] Nov 02 15:32:37.030529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:37.030493 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5aeac88e-6497d7f887-ptf2d"] Nov 02 15:32:37.963601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:37.963564 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" path="/var/lib/kubelet/pods/c0b4a363-0c44-454a-8c6d-fd5b059d4890/volumes" Nov 02 15:32:44.175944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.175900 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s"] Nov 02 15:32:44.176946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.176919 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerName="payload-processing" Nov 02 15:32:44.177068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.176950 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerName="payload-processing" Nov 02 15:32:44.177068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.177015 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8f0c7010-afcd-4ebe-abfb-9517ea0d0806" containerName="revoke-keys" Nov 02 15:32:44.177068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.177025 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0c7010-afcd-4ebe-abfb-9517ea0d0806" containerName="revoke-keys" Nov 02 15:32:44.177227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.177200 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b4a363-0c44-454a-8c6d-fd5b059d4890" containerName="payload-processing" Nov 02 15:32:44.177282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.177227 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0c7010-afcd-4ebe-abfb-9517ea0d0806" containerName="revoke-keys" Nov 02 15:32:44.182840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.182821 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.234670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.234645 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s"] Nov 02 15:32:44.275427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.275385 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:32:44.275767 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.275747 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:32:44.316845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.316817 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqpqx\" (UniqueName: \"kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.316969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.316869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.317067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.317045 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.418648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.418618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.418786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.418734 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.418874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.418862 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqpqx\" (UniqueName: \"kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.419253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.419224 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.421160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.421134 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.477470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.477396 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqpqx\" (UniqueName: \"kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx\") pod \"maas-api-revoke-keys-e2e-api-a-0de358-vc52s\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.496452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.496430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:44.876993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.876961 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s"] Nov 02 15:32:44.880705 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:32:44.880678 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858e0745_05ca_49b8_93cc_99c143755dc3.slice/crio-1471cca16072210aa9f1a15485927fab2b434d049ba5b9485989d065a332ebc5 WatchSource:0}: Error finding container 1471cca16072210aa9f1a15485927fab2b434d049ba5b9485989d065a332ebc5: Status 404 returned error can't find the container with id 1471cca16072210aa9f1a15485927fab2b434d049ba5b9485989d065a332ebc5 Nov 02 15:32:44.900967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:44.900940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" event={"ID":"858e0745-05ca-49b8-93cc-99c143755dc3","Type":"ContainerStarted","Data":"1471cca16072210aa9f1a15485927fab2b434d049ba5b9485989d065a332ebc5"} Nov 02 15:32:45.909341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:45.909307 5091 generic.go:356] "Generic (PLEG): container finished" podID="858e0745-05ca-49b8-93cc-99c143755dc3" containerID="d813821965d92cd11ef12e6f72a34045996d74c17030d11e8643fa127f57b540" exitCode=0 Nov 02 15:32:45.909742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:45.909380 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" event={"ID":"858e0745-05ca-49b8-93cc-99c143755dc3","Type":"ContainerDied","Data":"d813821965d92cd11ef12e6f72a34045996d74c17030d11e8643fa127f57b540"} Nov 02 15:32:47.068603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.068545 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:47.161595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.161555 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx\" (UniqueName: \"kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx\") pod \"858e0745-05ca-49b8-93cc-99c143755dc3\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " Nov 02 15:32:47.161781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.161656 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca\") pod \"858e0745-05ca-49b8-93cc-99c143755dc3\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " Nov 02 15:32:47.161861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.161843 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca\") pod \"858e0745-05ca-49b8-93cc-99c143755dc3\" (UID: \"858e0745-05ca-49b8-93cc-99c143755dc3\") " Nov 02 15:32:47.162177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.162063 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca" pod "858e0745-05ca-49b8-93cc-99c143755dc3" (UID: "858e0745-05ca-49b8-93cc-99c143755dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:32:47.162489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.162428 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:47.164533 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.164186 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca" pod "858e0745-05ca-49b8-93cc-99c143755dc3" (UID: "858e0745-05ca-49b8-93cc-99c143755dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:32:47.164533 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.164365 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx" pod "858e0745-05ca-49b8-93cc-99c143755dc3" (UID: "858e0745-05ca-49b8-93cc-99c143755dc3"). InnerVolumeSpecName "kube-api-access-fqpqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:32:47.263862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.263833 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/858e0745-05ca-49b8-93cc-99c143755dc3-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:47.263862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.263860 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqpqx\" (UniqueName: \"kubernetes.io/projected/858e0745-05ca-49b8-93cc-99c143755dc3-kube-api-access-fqpqx\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:32:47.923159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.923118 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" Nov 02 15:32:47.923159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.923150 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s" event={"ID":"858e0745-05ca-49b8-93cc-99c143755dc3","Type":"ContainerDied","Data":"1471cca16072210aa9f1a15485927fab2b434d049ba5b9485989d065a332ebc5"} Nov 02 15:32:47.923396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:32:47.923200 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1471cca16072210aa9f1a15485927fab2b434d049ba5b9485989d065a332ebc5" Nov 02 15:32:57.935474 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:32:57.935433 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:33:05.600430 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.600390 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l"] Nov 02 15:33:05.601032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.601017 5091 cpu_manager.go:414] "containerMap: removing container" podUID="858e0745-05ca-49b8-93cc-99c143755dc3" containerName="revoke-keys" Nov 02 15:33:05.601086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.601033 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="858e0745-05ca-49b8-93cc-99c143755dc3" containerName="revoke-keys" Nov 02 15:33:05.601155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.601146 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="858e0745-05ca-49b8-93cc-99c143755dc3" containerName="revoke-keys" Nov 02 15:33:05.604785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.604767 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.655783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.655746 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-adf096-serving-cert\"" Nov 02 15:33:05.706616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.706581 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l"] Nov 02 15:33:05.724609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.724574 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.724785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.724747 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.724930 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.724906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv855\" (UniqueName: \"kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.725032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.725013 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.826481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.826441 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv855\" (UniqueName: \"kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.826696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.826579 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.826696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.826627 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.826807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.826756 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.826914 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:33:05.826891 5091 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-serving-cert: secret "maas-api-e2e-shared-b-w1-adf096-serving-cert" not found Nov 02 15:33:05.826992 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:33:05.826979 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls podName:298339d2-4e94-4908-bee4-a9cae13936c8 nodeName:}" failed. No retries permitted until 2026-11-02 15:33:06.32696026 +0000 UTC m=+2140.491512795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls") pod "maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" (UID: "298339d2-4e94-4908-bee4-a9cae13936c8") : secret "maas-api-e2e-shared-b-w1-adf096-serving-cert" not found Nov 02 15:33:05.827467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.827444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.829440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.829415 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:05.885895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:05.885861 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv855\" (UniqueName: \"kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:06.339881 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:06.339787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:06.342379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:06.342345 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:06.518770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:06.518739 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:06.906232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:06.906201 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l"] Nov 02 15:33:06.910025 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:33:06.909996 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298339d2_4e94_4908_bee4_a9cae13936c8.slice/crio-cc933db19513d4a281e7df31f09290b47cce974b519a05a070a9069cc7b6227a WatchSource:0}: Error finding container cc933db19513d4a281e7df31f09290b47cce974b519a05a070a9069cc7b6227a: Status 404 returned error can't find the container with id cc933db19513d4a281e7df31f09290b47cce974b519a05a070a9069cc7b6227a Nov 02 15:33:07.073448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:07.073402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" event={"ID":"298339d2-4e94-4908-bee4-a9cae13936c8","Type":"ContainerStarted","Data":"cc933db19513d4a281e7df31f09290b47cce974b519a05a070a9069cc7b6227a"} Nov 02 15:33:08.083133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:08.083039 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" event={"ID":"298339d2-4e94-4908-bee4-a9cae13936c8","Type":"ContainerStarted","Data":"6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22"} Nov 02 15:33:08.083591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:08.083263 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:08.197685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:08.197626 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" podStartSLOduration=2.356619796 podStartE2EDuration="3.197610592s" totalImagesPullingTime="840.990796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:08.19518575 +0000 UTC m=+2142.359738291" watchObservedRunningTime="2026-11-02 15:33:08.197610592 +0000 UTC m=+2142.362163133" Nov 02 15:33:14.095718 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:14.095681 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:33:27.310532 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.310482 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r"] Nov 02 15:33:27.314740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.314720 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.371339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.371302 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r"] Nov 02 15:33:27.412239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.411757 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:33:27.412239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.411780 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:33:27.420288 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.420258 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmkf\" (UniqueName: \"kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.420419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.420320 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.420487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.420472 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.521489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.521457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmkf\" (UniqueName: \"kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.521683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.521503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.521683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.521579 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.522316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.522290 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.523851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.523833 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.580703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.580639 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmkf\" (UniqueName: \"kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf\") pod \"maas-api-revoke-sub-862da326-pk26r\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.628025 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:27.627995 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:27.940298 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:33:27.940229 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:33:28.018266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:28.018236 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r"] Nov 02 15:33:28.022367 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:33:28.022339 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12cd230_cfd0_4d6b_b8f3_2fe913b451a4.slice/crio-a9f23281076daa6ae06ff94a003a2ae70dbc6d8ce5ce21077f3f3bd51856e192 WatchSource:0}: Error finding container a9f23281076daa6ae06ff94a003a2ae70dbc6d8ce5ce21077f3f3bd51856e192: Status 404 returned error can't find the container with id a9f23281076daa6ae06ff94a003a2ae70dbc6d8ce5ce21077f3f3bd51856e192 Nov 02 15:33:28.254492 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:28.254455 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" event={"ID":"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4","Type":"ContainerStarted","Data":"a9f23281076daa6ae06ff94a003a2ae70dbc6d8ce5ce21077f3f3bd51856e192"} Nov 02 15:33:29.264648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:29.264606 5091 generic.go:356] "Generic (PLEG): container finished" podID="f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" containerID="447a3384bf7ed5248d62d0264797f0898197e5a262aea6a4af28ea3c14c55de0" exitCode=0 Nov 02 15:33:29.265126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:29.264749 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" event={"ID":"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4","Type":"ContainerDied","Data":"447a3384bf7ed5248d62d0264797f0898197e5a262aea6a4af28ea3c14c55de0"} Nov 02 15:33:30.445882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.445852 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:30.566866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.565013 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca\") pod \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " Nov 02 15:33:30.566866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.565255 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca\") pod \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " Nov 02 15:33:30.566866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.565427 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca" pod "f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" (UID: "f12cd230-cfd0-4d6b-b8f3-2fe913b451a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:30.566866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.565727 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf\" (UniqueName: \"kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf\") pod \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\" (UID: \"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4\") " Nov 02 15:33:30.571779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.570057 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:30.575670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.575640 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca" pod "f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" (UID: "f12cd230-cfd0-4d6b-b8f3-2fe913b451a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:33:30.576277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.575904 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf" pod "f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" (UID: "f12cd230-cfd0-4d6b-b8f3-2fe913b451a4"). InnerVolumeSpecName "kube-api-access-6cmkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:30.671304 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.671266 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:30.671304 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:30.671303 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cmkf\" (UniqueName: \"kubernetes.io/projected/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4-kube-api-access-6cmkf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:31.282405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:31.282370 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" Nov 02 15:33:31.282699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:31.282667 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r" event={"ID":"f12cd230-cfd0-4d6b-b8f3-2fe913b451a4","Type":"ContainerDied","Data":"a9f23281076daa6ae06ff94a003a2ae70dbc6d8ce5ce21077f3f3bd51856e192"} Nov 02 15:33:31.282824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:31.282723 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f23281076daa6ae06ff94a003a2ae70dbc6d8ce5ce21077f3f3bd51856e192" Nov 02 15:33:33.087098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:33.087062 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk"] Nov 02 15:33:33.087596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:33.087375 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" containerID="cri-o://17d110cd7f69f7f116896702f92b936e968b15e575e2d2444a29d51f17f2481b" gracePeriod=30 Nov 02 15:33:38.392442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.392138 5091 generic.go:356] "Generic (PLEG): container finished" podID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerID="17d110cd7f69f7f116896702f92b936e968b15e575e2d2444a29d51f17f2481b" exitCode=0 Nov 02 15:33:38.392442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.392331 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" event={"ID":"6b96e8e4-61a6-4f54-8021-772d390f04ec","Type":"ContainerDied","Data":"17d110cd7f69f7f116896702f92b936e968b15e575e2d2444a29d51f17f2481b"} Nov 02 15:33:38.494170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.494142 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:33:38.508231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508204 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508271 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508290 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508351 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508390 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508445 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508486 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508510 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508608 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo\") pod \"6b96e8e4-61a6-4f54-8021-772d390f04ec\" (UID: \"6b96e8e4-61a6-4f54-8021-772d390f04ec\") " Nov 02 15:33:38.508970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.508940 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:38.509172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.509150 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:38.509322 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.509293 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:38.509397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.509303 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:38.509966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.509946 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:38.510065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.510004 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.510130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.510065 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.510130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.510081 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.510130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.510095 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.511336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.511310 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:33:38.511793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.511763 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:33:38.511877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.511817 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "kube-api-access-fwlkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:38.511972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.511950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token" pod "6b96e8e4-61a6-4f54-8021-772d390f04ec" (UID: "6b96e8e4-61a6-4f54-8021-772d390f04ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:38.611779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.611748 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.611779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.611773 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b96e8e4-61a6-4f54-8021-772d390f04ec-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.611779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.611782 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.612013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.611791 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:38.612013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:38.611800 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlkp\" (UniqueName: \"kubernetes.io/projected/6b96e8e4-61a6-4f54-8021-772d390f04ec-kube-api-access-fwlkp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:39.401179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:39.401149 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" Nov 02 15:33:39.401770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:39.401173 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk" event={"ID":"6b96e8e4-61a6-4f54-8021-772d390f04ec","Type":"ContainerDied","Data":"cc2032774108827bab24ded0c134222de61501cc481af744509f144ca8a59d7b"} Nov 02 15:33:39.401770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:39.401250 5091 scope.go:122] "RemoveContainer" containerID="17d110cd7f69f7f116896702f92b936e968b15e575e2d2444a29d51f17f2481b" Nov 02 15:33:39.562608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:39.562573 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk"] Nov 02 15:33:39.566823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:39.566781 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-n5gnk"] Nov 02 15:33:39.963443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:39.963407 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" path="/var/lib/kubelet/pods/6b96e8e4-61a6-4f54-8021-772d390f04ec/volumes" Nov 02 15:33:44.443821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:44.443787 5091 generic.go:356] "Generic (PLEG): container finished" podID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerID="ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc" exitCode=0 Nov 02 15:33:44.444345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:44.443848 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" event={"ID":"e88f8840-f89d-4ec4-ae7c-a688205a452e","Type":"ContainerDied","Data":"ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc"} Nov 02 15:33:44.444345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:44.444239 5091 scope.go:122] "RemoveContainer" containerID="ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc" Nov 02 15:33:44.446271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:44.446228 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:33:44.446383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:44.446328 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:33:44.446383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:44.446372 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:33:45.456197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:45.456157 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" event={"ID":"e88f8840-f89d-4ec4-ae7c-a688205a452e","Type":"ContainerStarted","Data":"bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929"} Nov 02 15:33:45.651771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:45.651733 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:33:45.653271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:45.653242 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:33:45.653340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:45.653301 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 15:33:46.648539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:46.647805 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:33:46.648539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:46.647864 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 15:33:47.012336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.012258 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk"] Nov 02 15:33:47.012885 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.012870 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" Nov 02 15:33:47.012937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.012888 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" Nov 02 15:33:47.012937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.012929 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" containerName="revoke-keys" Nov 02 15:33:47.012937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.012934 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" containerName="revoke-keys" Nov 02 15:33:47.013052 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.013041 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b96e8e4-61a6-4f54-8021-772d390f04ec" containerName="istio-proxy" Nov 02 15:33:47.013086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.013062 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" containerName="revoke-keys" Nov 02 15:33:47.016686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.016668 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.072693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.072664 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk"] Nov 02 15:33:47.075664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.075631 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.075810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.075786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4dw5\" (UniqueName: \"kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.075871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.075834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.115124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.115096 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:33:47.115284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.115153 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:33:47.177316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.177280 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.177476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.177400 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4dw5\" (UniqueName: \"kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.177476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.177436 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.177925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.177905 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.180121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.180101 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.254363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.254328 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4dw5\" (UniqueName: \"kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.351591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.348746 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:47.643543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.643380 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:33:47.643543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.643439 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 15:33:47.763669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:47.763635 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk"] Nov 02 15:33:47.765636 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:33:47.765390 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4adad2a_d82a_4d8b_b0ce_925d6031e788.slice/crio-b23b32ea7efa0158c800ea916f02acad5c3e4b72145c36395db463aef647f530 WatchSource:0}: Error finding container b23b32ea7efa0158c800ea916f02acad5c3e4b72145c36395db463aef647f530: Status 404 returned error can't find the container with id b23b32ea7efa0158c800ea916f02acad5c3e4b72145c36395db463aef647f530 Nov 02 15:33:48.085999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:48.085969 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:33:48.490295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:48.490255 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" event={"ID":"b4adad2a-d82a-4d8b-b0ce-925d6031e788","Type":"ContainerStarted","Data":"379d0c4564617a84ee01078598b248065cc87cf36c89d4f1319c81958d4836e9"} Nov 02 15:33:48.490477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:48.490310 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" event={"ID":"b4adad2a-d82a-4d8b-b0ce-925d6031e788","Type":"ContainerStarted","Data":"b23b32ea7efa0158c800ea916f02acad5c3e4b72145c36395db463aef647f530"} Nov 02 15:33:48.610280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:48.610218 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" podStartSLOduration=2.610205446 podStartE2EDuration="2.610205446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:33:48.606867127 +0000 UTC m=+2182.771419680" watchObservedRunningTime="2026-11-02 15:33:48.610205446 +0000 UTC m=+2182.774757992" Nov 02 15:33:48.651543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:48.641881 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:33:48.651543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:48.641951 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 15:33:49.500968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:49.500937 5091 generic.go:356] "Generic (PLEG): container finished" podID="b4adad2a-d82a-4d8b-b0ce-925d6031e788" containerID="379d0c4564617a84ee01078598b248065cc87cf36c89d4f1319c81958d4836e9" exitCode=0 Nov 02 15:33:49.501355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:49.501026 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" event={"ID":"b4adad2a-d82a-4d8b-b0ce-925d6031e788","Type":"ContainerDied","Data":"379d0c4564617a84ee01078598b248065cc87cf36c89d4f1319c81958d4836e9"} Nov 02 15:33:49.653543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:49.649970 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 02 15:33:49.653543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:49.650028 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 02 15:33:50.657639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.657605 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:33:50.660752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.660727 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:33:50.689063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.689044 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:50.723233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.723208 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca\") pod \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " Nov 02 15:33:50.723393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.723335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca\") pod \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " Nov 02 15:33:50.723561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.723477 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5\" (UniqueName: \"kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5\") pod \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\" (UID: \"b4adad2a-d82a-4d8b-b0ce-925d6031e788\") " Nov 02 15:33:50.723789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.723762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca" pod "b4adad2a-d82a-4d8b-b0ce-925d6031e788" (UID: "b4adad2a-d82a-4d8b-b0ce-925d6031e788"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:33:50.724474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.724312 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:50.726223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.726198 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5" pod "b4adad2a-d82a-4d8b-b0ce-925d6031e788" (UID: "b4adad2a-d82a-4d8b-b0ce-925d6031e788"). InnerVolumeSpecName "kube-api-access-m4dw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:33:50.726370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.726315 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca" pod "b4adad2a-d82a-4d8b-b0ce-925d6031e788" (UID: "b4adad2a-d82a-4d8b-b0ce-925d6031e788"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:33:50.825299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.825227 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4adad2a-d82a-4d8b-b0ce-925d6031e788-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:50.825299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:50.825256 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4dw5\" (UniqueName: \"kubernetes.io/projected/b4adad2a-d82a-4d8b-b0ce-925d6031e788-kube-api-access-m4dw5\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:33:51.530554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:51.530481 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" Nov 02 15:33:51.532655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:51.530584 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk" event={"ID":"b4adad2a-d82a-4d8b-b0ce-925d6031e788","Type":"ContainerDied","Data":"b23b32ea7efa0158c800ea916f02acad5c3e4b72145c36395db463aef647f530"} Nov 02 15:33:51.532771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:51.532690 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23b32ea7efa0158c800ea916f02acad5c3e4b72145c36395db463aef647f530" Nov 02 15:33:53.915951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:53.915909 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl"] Nov 02 15:33:53.963369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:53.963329 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-6lmxl"] Nov 02 15:33:55.187493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:55.187449 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz"] Nov 02 15:33:55.234346 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:55.234309 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3602ca-2lzrz"] Nov 02 15:33:55.964784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:55.964746 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0c7010-afcd-4ebe-abfb-9517ea0d0806" path="/var/lib/kubelet/pods/8f0c7010-afcd-4ebe-abfb-9517ea0d0806/volumes" Nov 02 15:33:55.965433 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:33:55.965412 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e300c2af-2b85-4e4d-b982-78b5ca6196ea" path="/var/lib/kubelet/pods/e300c2af-2b85-4e4d-b982-78b5ca6196ea/volumes" Nov 02 15:33:57.937029 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:33:57.936985 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:34:08.647112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:08.647019 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s"] Nov 02 15:34:08.650766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:08.650737 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0de358-vc52s"] Nov 02 15:34:09.961049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:09.961016 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="858e0745-05ca-49b8-93cc-99c143755dc3" path="/var/lib/kubelet/pods/858e0745-05ca-49b8-93cc-99c143755dc3/volumes" Nov 02 15:34:13.381153 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.381113 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z"] Nov 02 15:34:13.381776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.381730 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b4adad2a-d82a-4d8b-b0ce-925d6031e788" containerName="revoke-keys" Nov 02 15:34:13.381776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.381743 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4adad2a-d82a-4d8b-b0ce-925d6031e788" containerName="revoke-keys" Nov 02 15:34:13.381904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.381897 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4adad2a-d82a-4d8b-b0ce-925d6031e788" containerName="revoke-keys" Nov 02 15:34:13.385793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.385770 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.437055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.437022 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-5afebc-openshift-default-dockercfg-57h7j\"" Nov 02 15:34:13.465426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465373 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.465638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465447 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.465704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465667 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjc47\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.465765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465738 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.465856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465835 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.465918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465900 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.465975 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465953 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.466033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.465988 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.466087 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.466046 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.490451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.490418 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z"] Nov 02 15:34:13.567016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.566982 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjc47\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567113 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567203 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567257 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567344 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.567728 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567704 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.569019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.567829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.569019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.568061 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.569019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.568350 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.569019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.568424 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.570229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.570203 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.571063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.571036 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.625876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.625850 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.626027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.626010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjc47\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47\") pod \"e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:13.712588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:13.711830 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:14.220669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.212672 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z"] Nov 02 15:34:14.273837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.273790 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.274093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.274075 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.274236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.274220 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:14.793060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.793025 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" event={"ID":"b61c5e58-10ce-4239-8f92-70f81bbcd7c6","Type":"ContainerStarted","Data":"fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0"} Nov 02 15:34:14.793485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.793091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" event={"ID":"b61c5e58-10ce-4239-8f92-70f81bbcd7c6","Type":"ContainerStarted","Data":"3a8aae516b055a07c844b854cd1de5913806c9d241dd7bfbf82250f6be703e06"} Nov 02 15:34:14.916403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:14.916347 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podStartSLOduration=1.9163320609999999 podStartE2EDuration="1.916332061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:14.91396739 +0000 UTC m=+2209.078519931" watchObservedRunningTime="2026-11-02 15:34:14.916332061 +0000 UTC m=+2209.080884601" Nov 02 15:34:15.712908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:15.712879 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:15.714414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:15.714381 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 15:34:15.714591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:15.714426 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 15:34:16.713485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:16.713452 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 15:34:16.713978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:16.713503 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 15:34:17.713415 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:17.713379 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 15:34:17.713606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:17.713431 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 15:34:18.713638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:18.713596 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 15:34:18.714120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:18.713660 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 15:34:19.713237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:19.713206 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 02 15:34:19.713414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:19.713258 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 02 15:34:20.185491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.185450 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd"] Nov 02 15:34:20.194907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.194879 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.295107 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd"] Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.297773 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.297817 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.297855 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgft7\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.297928 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.297959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.298030 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.298057 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.298098 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.298736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.298132 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.366609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.366541 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-822f77-openshift-default-dockercfg-nntrq\"" Nov 02 15:34:20.402587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.402531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.403214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.402844 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.403214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.402952 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.403214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.402984 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.403446 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403595 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403710 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403718 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403771 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.403985 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.404093 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgft7\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.404311 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.404636 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.407499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.407319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.408115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.407705 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.466582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.465971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgft7\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.467269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.467237 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token\") pod \"e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.508237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.507365 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb"] Nov 02 15:34:20.508237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.507740 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" containerID="cri-o://38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5" gracePeriod=30 Nov 02 15:34:20.527385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.523937 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:20.776376 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.776285 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:20.783562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.777811 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:20.783562 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:20.783468 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:34:21.068899 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.068436 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd"] Nov 02 15:34:21.075335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.075302 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:21.075437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.075373 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:21.075437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.075405 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:34:21.857247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.857206 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" event={"ID":"e8f984c6-f521-4ebd-8b59-602ebcbc3abd","Type":"ContainerStarted","Data":"1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970"} Nov 02 15:34:21.857772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.857270 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" event={"ID":"e8f984c6-f521-4ebd-8b59-602ebcbc3abd","Type":"ContainerStarted","Data":"9f2048856f0fe26d6bd0244537e7ae20e83f97fb1a1d53c635a98edf7b47e05a"} Nov 02 15:34:21.975906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:21.975855 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podStartSLOduration=1.975841694 podStartE2EDuration="1.975841694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:34:21.972650931 +0000 UTC m=+2216.137203483" watchObservedRunningTime="2026-11-02 15:34:21.975841694 +0000 UTC m=+2216.140394234" Nov 02 15:34:22.524891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:22.524856 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:22.530575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:22.530550 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:22.869991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:22.869958 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:22.870911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:22.870887 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:34:25.346012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.345969 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:34:25.346439 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.346041 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:34:25.822064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.822036 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:34:25.883712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.883685 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.883888 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.883753 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.883888 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.883863 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.883898 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.883950 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884008 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:25.884098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884016 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:25.884098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884081 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884174 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884233 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz\") pod \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\" (UID: \"c57a0c79-3d61-40ef-a0bd-5d8eea34454f\") " Nov 02 15:34:25.884349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884335 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:25.884453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884390 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:25.884644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884620 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:25.884872 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884853 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.884944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884873 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.884944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884883 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.884944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884892 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.884944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.884900 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.886294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.886265 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:25.886801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.886775 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:25.886801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.886789 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:25.886927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.886800 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz" pod "c57a0c79-3d61-40ef-a0bd-5d8eea34454f" (UID: "c57a0c79-3d61-40ef-a0bd-5d8eea34454f"). InnerVolumeSpecName "kube-api-access-kxzhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:25.894551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.894470 5091 generic.go:356] "Generic (PLEG): container finished" podID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerID="38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5" exitCode=0 Nov 02 15:34:25.894652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.894555 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" Nov 02 15:34:25.894652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.894551 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" event={"ID":"c57a0c79-3d61-40ef-a0bd-5d8eea34454f","Type":"ContainerDied","Data":"38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5"} Nov 02 15:34:25.894759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.894682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb" event={"ID":"c57a0c79-3d61-40ef-a0bd-5d8eea34454f","Type":"ContainerDied","Data":"7d3d7454901563abbc8d2512787d547aec669ebd68e120dd7c5f03b4e3ab5db0"} Nov 02 15:34:25.894759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.894710 5091 scope.go:122] "RemoveContainer" containerID="38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5" Nov 02 15:34:25.916751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.916734 5091 scope.go:122] "RemoveContainer" containerID="38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5" Nov 02 15:34:25.917027 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:34:25.917008 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5\": container with ID starting with 38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5 not found: ID does not exist" containerID="38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5" Nov 02 15:34:25.917079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.917034 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5"} err="failed to get container status \"38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5\": rpc error: code = NotFound desc = could not find container \"38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5\": container with ID starting with 38d9c6ed25cebb7f919404bdf52c6b7e4d96e1f2769f5ba0e1cb175c85b598b5 not found: ID does not exist" Nov 02 15:34:25.986126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.986098 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxzhz\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-kube-api-access-kxzhz\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.986126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.986124 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.986324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.986134 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:25.986324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:25.986145 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c57a0c79-3d61-40ef-a0bd-5d8eea34454f-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:26.100068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:26.100011 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb"] Nov 02 15:34:26.104355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:26.104328 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-nvbgb"] Nov 02 15:34:27.936061 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:34:27.936017 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:34:27.964467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:27.964425 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" path="/var/lib/kubelet/pods/c57a0c79-3d61-40ef-a0bd-5d8eea34454f/volumes" Nov 02 15:34:31.938882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:31.938850 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg"] Nov 02 15:34:31.939294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:31.939046 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" containerID="cri-o://d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226" gracePeriod=30 Nov 02 15:34:32.964650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:32.964606 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:34:32.965044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:32.964685 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:34:37.236278 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.236249 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:34:37.280016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.279978 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280009 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280085 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280151 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280305 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:37.280340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280328 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280385 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280420 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280472 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket\") pod \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\" (UID: \"5fe428c1-d9b9-4c39-9232-6faa98e597d7\") " Nov 02 15:34:37.280699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280483 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:37.280760 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280731 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:37.280819 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280759 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:34:37.280985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.280959 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:37.281068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.281034 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.281068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.281051 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.281068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.281061 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.281231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.281070 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.282359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.282337 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:34:37.282764 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.282740 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:37.282879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.282746 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:34:37.282943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.282908 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq" pod "5fe428c1-d9b9-4c39-9232-6faa98e597d7" (UID: "5fe428c1-d9b9-4c39-9232-6faa98e597d7"). InnerVolumeSpecName "kube-api-access-25ddq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:34:37.382435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.382407 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.382435 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.382437 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.382616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.382451 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.382616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.382465 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:37.382616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:37.382478 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25ddq\" (UniqueName: \"kubernetes.io/projected/5fe428c1-d9b9-4c39-9232-6faa98e597d7-kube-api-access-25ddq\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:34:38.017188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.017149 5091 generic.go:356] "Generic (PLEG): container finished" podID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerID="d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226" exitCode=0 Nov 02 15:34:38.017352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.017214 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" Nov 02 15:34:38.017352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.017229 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" event={"ID":"5fe428c1-d9b9-4c39-9232-6faa98e597d7","Type":"ContainerDied","Data":"d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226"} Nov 02 15:34:38.017352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.017287 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg" event={"ID":"5fe428c1-d9b9-4c39-9232-6faa98e597d7","Type":"ContainerDied","Data":"98030b5b9f4f5db969a982996f5d6ea63c8adcb3ec952b973668d7eb0cf3aa89"} Nov 02 15:34:38.017352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.017309 5091 scope.go:122] "RemoveContainer" containerID="d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226" Nov 02 15:34:38.027687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.027666 5091 scope.go:122] "RemoveContainer" containerID="d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226" Nov 02 15:34:38.027969 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:34:38.027948 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226\": container with ID starting with d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226 not found: ID does not exist" containerID="d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226" Nov 02 15:34:38.028022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.027978 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226"} err="failed to get container status \"d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226\": rpc error: code = NotFound desc = could not find container \"d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226\": container with ID starting with d6cf168bf99035502ffadba793b192af27e4f6445f0dedbd90c0f6426dc80226 not found: ID does not exist" Nov 02 15:34:38.177125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.177083 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg"] Nov 02 15:34:38.224448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:38.224371 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-vb7bg"] Nov 02 15:34:39.984541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:34:39.984015 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" path="/var/lib/kubelet/pods/5fe428c1-d9b9-4c39-9232-6faa98e597d7/volumes" Nov 02 15:34:57.936330 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:34:57.936291 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:35:09.778588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:35:09.778545 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk"] Nov 02 15:35:09.823366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:35:09.823332 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-941182-8dppk"] Nov 02 15:35:09.964883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:35:09.964844 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4adad2a-d82a-4d8b-b0ce-925d6031e788" path="/var/lib/kubelet/pods/b4adad2a-d82a-4d8b-b0ce-925d6031e788/volumes" Nov 02 15:35:27.938849 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:35:27.938811 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:35:57.939164 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:35:57.939069 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:36:01.252018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.251978 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw"] Nov 02 15:36:01.252627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.252613 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" Nov 02 15:36:01.252627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.252627 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" Nov 02 15:36:01.252721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.252644 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" Nov 02 15:36:01.252721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.252649 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" Nov 02 15:36:01.252784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.252736 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57a0c79-3d61-40ef-a0bd-5d8eea34454f" containerName="istio-proxy" Nov 02 15:36:01.252784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.252753 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe428c1-d9b9-4c39-9232-6faa98e597d7" containerName="istio-proxy" Nov 02 15:36:01.256567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.256548 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.271808 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.271779 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpxp\" (UniqueName: \"kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.272057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.272024 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.272168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.272086 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.354298 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.354255 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:36:01.354298 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.354259 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:36:01.358498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.358473 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw"] Nov 02 15:36:01.373154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.373126 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpxp\" (UniqueName: \"kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.373291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.373256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.373365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.373306 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.374074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.374050 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.375787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.375764 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.431423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.431392 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpxp\" (UniqueName: \"kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.573383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.573300 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:01.952373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:01.952340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw"] Nov 02 15:36:01.956630 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:36:01.956597 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43daa66_75ba_4e58_87c0_a8caa6c909af.slice/crio-71cad8b1fb365520888010b7f3e892623576c953bca003f2912347cf64bb3bd1 WatchSource:0}: Error finding container 71cad8b1fb365520888010b7f3e892623576c953bca003f2912347cf64bb3bd1: Status 404 returned error can't find the container with id 71cad8b1fb365520888010b7f3e892623576c953bca003f2912347cf64bb3bd1 Nov 02 15:36:02.826718 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:02.826674 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" event={"ID":"a43daa66-75ba-4e58-87c0-a8caa6c909af","Type":"ContainerStarted","Data":"ba4ac51bc7624c3f11ab332a6ed0c159453e9a2c315d8e938e758170f77bd1e7"} Nov 02 15:36:02.827202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:02.826739 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" event={"ID":"a43daa66-75ba-4e58-87c0-a8caa6c909af","Type":"ContainerStarted","Data":"71cad8b1fb365520888010b7f3e892623576c953bca003f2912347cf64bb3bd1"} Nov 02 15:36:02.938633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:02.938582 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" podStartSLOduration=1.938568433 podStartE2EDuration="1.938568433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:02.936852581 +0000 UTC m=+2317.101405122" watchObservedRunningTime="2026-11-02 15:36:02.938568433 +0000 UTC m=+2317.103120972" Nov 02 15:36:03.838474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:03.838432 5091 generic.go:356] "Generic (PLEG): container finished" podID="a43daa66-75ba-4e58-87c0-a8caa6c909af" containerID="ba4ac51bc7624c3f11ab332a6ed0c159453e9a2c315d8e938e758170f77bd1e7" exitCode=0 Nov 02 15:36:03.838991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:03.838545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" event={"ID":"a43daa66-75ba-4e58-87c0-a8caa6c909af","Type":"ContainerDied","Data":"ba4ac51bc7624c3f11ab332a6ed0c159453e9a2c315d8e938e758170f77bd1e7"} Nov 02 15:36:05.003602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.003576 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:05.122695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.122667 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp\" (UniqueName: \"kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp\") pod \"a43daa66-75ba-4e58-87c0-a8caa6c909af\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " Nov 02 15:36:05.122880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.122835 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca\") pod \"a43daa66-75ba-4e58-87c0-a8caa6c909af\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " Nov 02 15:36:05.122931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.122899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca\") pod \"a43daa66-75ba-4e58-87c0-a8caa6c909af\" (UID: \"a43daa66-75ba-4e58-87c0-a8caa6c909af\") " Nov 02 15:36:05.123942 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.123729 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca" pod "a43daa66-75ba-4e58-87c0-a8caa6c909af" (UID: "a43daa66-75ba-4e58-87c0-a8caa6c909af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:05.127234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.125183 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp" pod "a43daa66-75ba-4e58-87c0-a8caa6c909af" (UID: "a43daa66-75ba-4e58-87c0-a8caa6c909af"). InnerVolumeSpecName "kube-api-access-7gpxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:05.131913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.131888 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca" pod "a43daa66-75ba-4e58-87c0-a8caa6c909af" (UID: "a43daa66-75ba-4e58-87c0-a8caa6c909af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:05.224695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.224661 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.224695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.224691 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43daa66-75ba-4e58-87c0-a8caa6c909af-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.224882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.224703 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpxp\" (UniqueName: \"kubernetes.io/projected/a43daa66-75ba-4e58-87c0-a8caa6c909af-kube-api-access-7gpxp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:05.867249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.867218 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" Nov 02 15:36:05.867638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.867608 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw" event={"ID":"a43daa66-75ba-4e58-87c0-a8caa6c909af","Type":"ContainerDied","Data":"71cad8b1fb365520888010b7f3e892623576c953bca003f2912347cf64bb3bd1"} Nov 02 15:36:05.867823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:05.867806 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71cad8b1fb365520888010b7f3e892623576c953bca003f2912347cf64bb3bd1" Nov 02 15:36:22.871967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:22.871934 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:22.872511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:22.871987 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:23.017591 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:23.017558 5091 generic.go:356] "Generic (PLEG): container finished" podID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerID="1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970" exitCode=0 Nov 02 15:36:23.017773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:23.017627 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" event={"ID":"e8f984c6-f521-4ebd-8b59-602ebcbc3abd","Type":"ContainerDied","Data":"1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970"} Nov 02 15:36:23.018034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:23.018016 5091 scope.go:122] "RemoveContainer" containerID="1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970" Nov 02 15:36:23.019672 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:23.019648 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:23.019735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:23.019715 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:23.019771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:23.019747 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:24.026976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.026922 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" event={"ID":"e8f984c6-f521-4ebd-8b59-602ebcbc3abd","Type":"ContainerStarted","Data":"37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b"} Nov 02 15:36:24.412262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.412227 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4"] Nov 02 15:36:24.412894 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.412879 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a43daa66-75ba-4e58-87c0-a8caa6c909af" containerName="revoke-keys" Nov 02 15:36:24.412966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.412895 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43daa66-75ba-4e58-87c0-a8caa6c909af" containerName="revoke-keys" Nov 02 15:36:24.413003 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.412988 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43daa66-75ba-4e58-87c0-a8caa6c909af" containerName="revoke-keys" Nov 02 15:36:24.416571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.416555 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.501542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.501497 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4"] Nov 02 15:36:24.524837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.524809 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:36:24.524972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.524952 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:24.525014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.524996 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:24.550856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.550820 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5aeac88e-openshift-default-dockercfg-hfkbc\"" Nov 02 15:36:24.573372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.573325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.573709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.573668 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.573982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.573954 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.574196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.574178 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.574426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.574405 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.574653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.574635 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p744c\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.574840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.574811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.574995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.574979 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.575159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.575142 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676249 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676301 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676615 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676615 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676456 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676615 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676585 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676663 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p744c\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.676988 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.676961 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677086 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677199 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677014 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677210 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677210 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677142 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.677825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.677383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.682890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.679579 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.682890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.680058 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.752549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.751558 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p744c\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:24.752549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:24.752117 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:25.043589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.039745 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:25.547231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.534923 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:25.547231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.534994 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:25.669035 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.668987 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4"] Nov 02 15:36:25.721200 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.721161 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:25.721384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.721366 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:25.721725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:25.721703 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:26.149543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:26.135698 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" event={"ID":"7c9cea57-bd65-4de9-b951-662b2347149a","Type":"ContainerStarted","Data":"57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b"} Nov 02 15:36:26.149543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:26.135793 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" event={"ID":"7c9cea57-bd65-4de9-b951-662b2347149a","Type":"ContainerStarted","Data":"2c51b7ea0c628f955ccadbcb5c3c04dadaaa404f38c4aeac7830da52967204d6"} Nov 02 15:36:26.269544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:26.267962 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podStartSLOduration=2.267947047 podStartE2EDuration="2.267947047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:26.26431601 +0000 UTC m=+2340.428868551" watchObservedRunningTime="2026-11-02 15:36:26.267947047 +0000 UTC m=+2340.432499586" Nov 02 15:36:26.524841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:26.524761 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:26.524841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:26.524817 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:27.043540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:27.042918 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:27.061544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:27.057676 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:27.061544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:27.057768 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 15:36:27.524743 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:27.524707 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:27.525225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:27.524766 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:28.000893 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:36:28.000710 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:36:28.041344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:28.040891 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:28.041344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:28.040984 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 15:36:28.524458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:28.524420 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:28.524671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:28.524485 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:29.045917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:29.045878 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:29.046651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:29.046619 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 15:36:29.525170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:29.525125 5091 patch_prober.go:29] interesting pod/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Nov 02 15:36:29.525339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:29.525188 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Nov 02 15:36:30.041540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:30.040938 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:30.041540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:30.040998 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 15:36:30.546541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:30.535950 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:36:30.546541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:30.537396 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:36:30.546541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:30.538704 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:36:31.048927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:31.048889 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:31.049206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:31.049175 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 15:36:32.040965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:32.040925 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Nov 02 15:36:32.041592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:32.041558 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Nov 02 15:36:33.045602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:33.045571 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:33.046056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:33.046035 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:33.046831 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:33.046813 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:36:35.189638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.189595 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r"] Nov 02 15:36:35.195031 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.195011 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.248866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.248841 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r"] Nov 02 15:36:35.258370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.258342 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.258511 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.258388 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.258604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.258579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wg87\" (UniqueName: \"kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.291079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.291045 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:36:35.291079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.291055 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:36:35.359891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.359859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.360035 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.359904 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.360035 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.360016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wg87\" (UniqueName: \"kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.360561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.360532 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.362414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.362388 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.418284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.418258 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wg87\" (UniqueName: \"kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.510310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.510243 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:35.887839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:35.887809 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r"] Nov 02 15:36:35.892288 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:36:35.892259 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1f4705_17ed_4447_8c31_3baedc0c4f07.slice/crio-7bdd53550483d87dc25c918395dd3f6d50e094c0d9fa1d77d72cfed96e47f496 WatchSource:0}: Error finding container 7bdd53550483d87dc25c918395dd3f6d50e094c0d9fa1d77d72cfed96e47f496: Status 404 returned error can't find the container with id 7bdd53550483d87dc25c918395dd3f6d50e094c0d9fa1d77d72cfed96e47f496 Nov 02 15:36:36.250494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:36.250464 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" event={"ID":"1f1f4705-17ed-4447-8c31-3baedc0c4f07","Type":"ContainerStarted","Data":"d953057faebe197df3c9ac7c467b5b9f78b5e25f3cd64aab1a91e41afda37cc3"} Nov 02 15:36:36.250883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:36.250542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" event={"ID":"1f1f4705-17ed-4447-8c31-3baedc0c4f07","Type":"ContainerStarted","Data":"7bdd53550483d87dc25c918395dd3f6d50e094c0d9fa1d77d72cfed96e47f496"} Nov 02 15:36:36.364191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:36.364139 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" podStartSLOduration=1.364125023 podStartE2EDuration="1.364125023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:36.361650614 +0000 UTC m=+2350.526203166" watchObservedRunningTime="2026-11-02 15:36:36.364125023 +0000 UTC m=+2350.528677562" Nov 02 15:36:37.260928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:37.260893 5091 generic.go:356] "Generic (PLEG): container finished" podID="1f1f4705-17ed-4447-8c31-3baedc0c4f07" containerID="d953057faebe197df3c9ac7c467b5b9f78b5e25f3cd64aab1a91e41afda37cc3" exitCode=0 Nov 02 15:36:37.261392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:37.260966 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" event={"ID":"1f1f4705-17ed-4447-8c31-3baedc0c4f07","Type":"ContainerDied","Data":"d953057faebe197df3c9ac7c467b5b9f78b5e25f3cd64aab1a91e41afda37cc3"} Nov 02 15:36:38.472353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.472328 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:38.502654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.502619 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca\") pod \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " Nov 02 15:36:38.502795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.502758 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87\" (UniqueName: \"kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87\") pod \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " Nov 02 15:36:38.502865 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.502797 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca\") pod \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\" (UID: \"1f1f4705-17ed-4447-8c31-3baedc0c4f07\") " Nov 02 15:36:38.503259 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.503218 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca" pod "1f1f4705-17ed-4447-8c31-3baedc0c4f07" (UID: "1f1f4705-17ed-4447-8c31-3baedc0c4f07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:38.503567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.503545 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:38.504948 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.504919 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca" pod "1f1f4705-17ed-4447-8c31-3baedc0c4f07" (UID: "1f1f4705-17ed-4447-8c31-3baedc0c4f07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:38.505044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.504960 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87" pod "1f1f4705-17ed-4447-8c31-3baedc0c4f07" (UID: "1f1f4705-17ed-4447-8c31-3baedc0c4f07"). InnerVolumeSpecName "kube-api-access-9wg87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:38.604799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.604702 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1f4705-17ed-4447-8c31-3baedc0c4f07-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:38.604799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:38.604741 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wg87\" (UniqueName: \"kubernetes.io/projected/1f1f4705-17ed-4447-8c31-3baedc0c4f07-kube-api-access-9wg87\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:39.278468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.278437 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" Nov 02 15:36:39.278468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.278446 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r" event={"ID":"1f1f4705-17ed-4447-8c31-3baedc0c4f07","Type":"ContainerDied","Data":"7bdd53550483d87dc25c918395dd3f6d50e094c0d9fa1d77d72cfed96e47f496"} Nov 02 15:36:39.278709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.278497 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bdd53550483d87dc25c918395dd3f6d50e094c0d9fa1d77d72cfed96e47f496" Nov 02 15:36:39.548440 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.548353 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj"] Nov 02 15:36:39.549022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.549005 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1f1f4705-17ed-4447-8c31-3baedc0c4f07" containerName="revoke-keys" Nov 02 15:36:39.549085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.549023 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1f4705-17ed-4447-8c31-3baedc0c4f07" containerName="revoke-keys" Nov 02 15:36:39.549152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.549142 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1f4705-17ed-4447-8c31-3baedc0c4f07" containerName="revoke-keys" Nov 02 15:36:39.553015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.552997 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.618826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.618786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.618928 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619069 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7mw\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619368 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619430 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619535 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.619622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.619590 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.634416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.634381 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj"] Nov 02 15:36:39.720792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.720758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.720989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.720828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.720989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.720896 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.720989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.720917 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.720989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.720947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7mw\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721010 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721070 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721106 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721221 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721291 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721509 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721463 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721631 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.721916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.721893 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.723550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.723526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.724202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.724176 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.779484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.779449 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.779782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.779757 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7mw\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw\") pod \"e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:39.869704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:39.869672 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:40.309472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:40.309391 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj"] Nov 02 15:36:40.313384 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:36:40.313357 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f3f38d_1735_47ff_a556_05cae9593a0f.slice/crio-b4186ee4cbfdb2f8a19f92f4861938b8dba42791c62a726da7fa273cf5a6cfda WatchSource:0}: Error finding container b4186ee4cbfdb2f8a19f92f4861938b8dba42791c62a726da7fa273cf5a6cfda: Status 404 returned error can't find the container with id b4186ee4cbfdb2f8a19f92f4861938b8dba42791c62a726da7fa273cf5a6cfda Nov 02 15:36:40.316093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:40.316062 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.316167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:40.316128 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:40.316167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:40.316157 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:36:41.295795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:41.295756 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" event={"ID":"e8f3f38d-1735-47ff-a556-05cae9593a0f","Type":"ContainerStarted","Data":"fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63"} Nov 02 15:36:41.296259 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:41.295823 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" event={"ID":"e8f3f38d-1735-47ff-a556-05cae9593a0f","Type":"ContainerStarted","Data":"b4186ee4cbfdb2f8a19f92f4861938b8dba42791c62a726da7fa273cf5a6cfda"} Nov 02 15:36:41.461510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:41.461458 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podStartSLOduration=2.46144305 podStartE2EDuration="2.46144305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:36:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:36:41.459201902 +0000 UTC m=+2355.623754440" watchObservedRunningTime="2026-11-02 15:36:41.46144305 +0000 UTC m=+2355.625995668" Nov 02 15:36:41.870186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:41.870148 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:41.871560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:41.871529 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:36:41.871671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:41.871589 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 15:36:42.870170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:42.870131 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:36:42.870610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:42.870202 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 15:36:43.871123 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:43.871085 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:36:43.871580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:43.871183 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 15:36:44.871102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:44.871059 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:36:44.871370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:44.871119 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 15:36:45.871138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:45.871091 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 02 15:36:45.871349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:45.871164 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 02 15:36:46.875003 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:46.874970 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:46.875512 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:46.875488 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:46.876230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:46.876210 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:36:48.142406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.142366 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq"] Nov 02 15:36:48.142913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.142664 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" podUID="56a78bf5-2886-4938-b220-f4775b170a36" containerName="maas-api" containerID="cri-o://7b903a95e55bb714445bd3bfa805da5ebc49fe1926dd3cc0de6f544cf2d1c875" gracePeriod=30 Nov 02 15:36:48.432645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.432187 5091 generic.go:356] "Generic (PLEG): container finished" podID="56a78bf5-2886-4938-b220-f4775b170a36" containerID="7b903a95e55bb714445bd3bfa805da5ebc49fe1926dd3cc0de6f544cf2d1c875" exitCode=0 Nov 02 15:36:48.433642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.433574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" event={"ID":"56a78bf5-2886-4938-b220-f4775b170a36","Type":"ContainerDied","Data":"7b903a95e55bb714445bd3bfa805da5ebc49fe1926dd3cc0de6f544cf2d1c875"} Nov 02 15:36:48.499089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.499060 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:36:48.628983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.628944 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\") pod \"56a78bf5-2886-4938-b220-f4775b170a36\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " Nov 02 15:36:48.629308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.629283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg\" (UniqueName: \"kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg\") pod \"56a78bf5-2886-4938-b220-f4775b170a36\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " Nov 02 15:36:48.629487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.629472 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs\") pod \"56a78bf5-2886-4938-b220-f4775b170a36\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " Nov 02 15:36:48.629759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.629742 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca\") pod \"56a78bf5-2886-4938-b220-f4775b170a36\" (UID: \"56a78bf5-2886-4938-b220-f4775b170a36\") " Nov 02 15:36:48.637279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.637230 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs" pod "56a78bf5-2886-4938-b220-f4775b170a36" (UID: "56a78bf5-2886-4938-b220-f4775b170a36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:48.637710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.637686 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca" pod "56a78bf5-2886-4938-b220-f4775b170a36" (UID: "56a78bf5-2886-4938-b220-f4775b170a36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:36:48.642820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.642781 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls" pod "56a78bf5-2886-4938-b220-f4775b170a36" (UID: "56a78bf5-2886-4938-b220-f4775b170a36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:36:48.643049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.643030 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg" pod "56a78bf5-2886-4938-b220-f4775b170a36" (UID: "56a78bf5-2886-4938-b220-f4775b170a36"). InnerVolumeSpecName "kube-api-access-wk4tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:36:48.736307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.736212 5091 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a78bf5-2886-4938-b220-f4775b170a36-openshift-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:48.736307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.736244 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-maas-api-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:48.736307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.736254 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk4tg\" (UniqueName: \"kubernetes.io/projected/56a78bf5-2886-4938-b220-f4775b170a36-kube-api-access-wk4tg\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:48.736307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:48.736264 5091 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56a78bf5-2886-4938-b220-f4775b170a36-metrics-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:36:49.441925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:49.441895 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" Nov 02 15:36:49.444150 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:49.441888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq" event={"ID":"56a78bf5-2886-4938-b220-f4775b170a36","Type":"ContainerDied","Data":"8c00829025ac48d3e42be825d61e17b44a54ec6912e268a1a3023743befd2807"} Nov 02 15:36:49.444150 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:49.442078 5091 scope.go:122] "RemoveContainer" containerID="7b903a95e55bb714445bd3bfa805da5ebc49fe1926dd3cc0de6f544cf2d1c875" Nov 02 15:36:49.602271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:49.602234 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq"] Nov 02 15:36:49.648674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:49.648635 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-fb80ca-b988b5dd7-k65bq"] Nov 02 15:36:49.960999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:49.960965 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a78bf5-2886-4938-b220-f4775b170a36" path="/var/lib/kubelet/pods/56a78bf5-2886-4938-b220-f4775b170a36/volumes" Nov 02 15:36:50.049915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:50.049883 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm"] Nov 02 15:36:50.053321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:50.053296 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c96e6963-z29rm"] Nov 02 15:36:51.962698 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:36:51.962664 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185dc6e4-d74e-44c1-a5df-101e897a00cb" path="/var/lib/kubelet/pods/185dc6e4-d74e-44c1-a5df-101e897a00cb/volumes" Nov 02 15:36:57.999041 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:36:57.998995 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:37:01.279774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.277343 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn"] Nov 02 15:37:01.279774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.278921 5091 cpu_manager.go:414] "containerMap: removing container" podUID="56a78bf5-2886-4938-b220-f4775b170a36" containerName="maas-api" Nov 02 15:37:01.279774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.278946 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a78bf5-2886-4938-b220-f4775b170a36" containerName="maas-api" Nov 02 15:37:01.279774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.279375 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a78bf5-2886-4938-b220-f4775b170a36" containerName="maas-api" Nov 02 15:37:01.286771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.286650 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.326970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.326940 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.327209 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.327181 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fl6\" (UniqueName: \"kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.327265 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.327242 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.335353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.335329 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn"] Nov 02 15:37:01.377528 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.377486 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:37:01.377836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.377816 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:37:01.428211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.428172 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.428369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.428346 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fl6\" (UniqueName: \"kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.428432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.428402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.428859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.428839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.430856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.430831 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.487210 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.487184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fl6\" (UniqueName: \"kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6\") pod \"maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.600692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.600621 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:01.989237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:01.989201 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn"] Nov 02 15:37:01.993532 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:37:01.993487 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c069f5f_a487_4791_b786_de9d0e7cf520.slice/crio-2ea93b686b81e61a06fb9f6aedbb31772ff25811fc4efc4f79a273bbe29269c0 WatchSource:0}: Error finding container 2ea93b686b81e61a06fb9f6aedbb31772ff25811fc4efc4f79a273bbe29269c0: Status 404 returned error can't find the container with id 2ea93b686b81e61a06fb9f6aedbb31772ff25811fc4efc4f79a273bbe29269c0 Nov 02 15:37:02.614594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:02.603056 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" event={"ID":"8c069f5f-a487-4791-b786-de9d0e7cf520","Type":"ContainerStarted","Data":"5497c8caa21f8664fb0d517442ddb962a2b9bebb2b4f008a55a6df1147e612af"} Nov 02 15:37:02.614594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:02.603136 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" event={"ID":"8c069f5f-a487-4791-b786-de9d0e7cf520","Type":"ContainerStarted","Data":"2ea93b686b81e61a06fb9f6aedbb31772ff25811fc4efc4f79a273bbe29269c0"} Nov 02 15:37:02.726659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:02.726582 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" podStartSLOduration=1.726560995 podStartE2EDuration="1.726560995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:02.72086582 +0000 UTC m=+2376.885418361" watchObservedRunningTime="2026-11-02 15:37:02.726560995 +0000 UTC m=+2376.891113536" Nov 02 15:37:03.612118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:03.612078 5091 generic.go:356] "Generic (PLEG): container finished" podID="8c069f5f-a487-4791-b786-de9d0e7cf520" containerID="5497c8caa21f8664fb0d517442ddb962a2b9bebb2b4f008a55a6df1147e612af" exitCode=0 Nov 02 15:37:03.612297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:03.612140 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" event={"ID":"8c069f5f-a487-4791-b786-de9d0e7cf520","Type":"ContainerDied","Data":"5497c8caa21f8664fb0d517442ddb962a2b9bebb2b4f008a55a6df1147e612af"} Nov 02 15:37:04.794716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.794672 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:04.905664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.905584 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6\" (UniqueName: \"kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6\") pod \"8c069f5f-a487-4791-b786-de9d0e7cf520\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " Nov 02 15:37:04.905822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.905701 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca\") pod \"8c069f5f-a487-4791-b786-de9d0e7cf520\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " Nov 02 15:37:04.905822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.905775 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca\") pod \"8c069f5f-a487-4791-b786-de9d0e7cf520\" (UID: \"8c069f5f-a487-4791-b786-de9d0e7cf520\") " Nov 02 15:37:04.906814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.906782 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca" pod "8c069f5f-a487-4791-b786-de9d0e7cf520" (UID: "8c069f5f-a487-4791-b786-de9d0e7cf520"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:04.909506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.909296 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6" pod "8c069f5f-a487-4791-b786-de9d0e7cf520" (UID: "8c069f5f-a487-4791-b786-de9d0e7cf520"). InnerVolumeSpecName "kube-api-access-h2fl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:04.929306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:04.929274 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca" pod "8c069f5f-a487-4791-b786-de9d0e7cf520" (UID: "8c069f5f-a487-4791-b786-de9d0e7cf520"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:05.008479 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:05.008445 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:05.008479 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:05.008478 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c069f5f-a487-4791-b786-de9d0e7cf520-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:05.008660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:05.008493 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2fl6\" (UniqueName: \"kubernetes.io/projected/8c069f5f-a487-4791-b786-de9d0e7cf520-kube-api-access-h2fl6\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:05.627738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:05.627705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" event={"ID":"8c069f5f-a487-4791-b786-de9d0e7cf520","Type":"ContainerDied","Data":"2ea93b686b81e61a06fb9f6aedbb31772ff25811fc4efc4f79a273bbe29269c0"} Nov 02 15:37:05.627914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:05.627755 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea93b686b81e61a06fb9f6aedbb31772ff25811fc4efc4f79a273bbe29269c0" Nov 02 15:37:05.627914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:05.627718 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn" Nov 02 15:37:09.468846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.468767 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj"] Nov 02 15:37:09.469612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.469590 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8c069f5f-a487-4791-b786-de9d0e7cf520" containerName="revoke-keys" Nov 02 15:37:09.469612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.469611 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c069f5f-a487-4791-b786-de9d0e7cf520" containerName="revoke-keys" Nov 02 15:37:09.469746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.469737 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c069f5f-a487-4791-b786-de9d0e7cf520" containerName="revoke-keys" Nov 02 15:37:09.473787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.473768 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.491360 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.491332 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.491456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.491378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.491497 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.491469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldl2r\" (UniqueName: \"kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.523890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.523865 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:37:09.526097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.526073 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj"] Nov 02 15:37:09.569023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.568995 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:37:09.593298 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.593272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.593420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.593314 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.593420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.593339 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldl2r\" (UniqueName: \"kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.594063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.594041 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.595686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.595662 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.651684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.651655 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldl2r\" (UniqueName: \"kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r\") pod \"maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:09.788577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:09.788485 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:10.174405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:10.174370 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj"] Nov 02 15:37:10.176234 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:37:10.176191 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7e188c_0a25_49be_a5e3_fb705bc86a91.slice/crio-1650a03b0cc3f10227279720fad32aa12af011494cd3b71a7c7ca34811236aa4 WatchSource:0}: Error finding container 1650a03b0cc3f10227279720fad32aa12af011494cd3b71a7c7ca34811236aa4: Status 404 returned error can't find the container with id 1650a03b0cc3f10227279720fad32aa12af011494cd3b71a7c7ca34811236aa4 Nov 02 15:37:10.672166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:10.672125 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" event={"ID":"dd7e188c-0a25-49be-a5e3-fb705bc86a91","Type":"ContainerStarted","Data":"24675509e1760a3b06621e81d0581b815a32f0838eb8ef3620b65bfc4c10c6af"} Nov 02 15:37:10.672599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:10.672213 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" event={"ID":"dd7e188c-0a25-49be-a5e3-fb705bc86a91","Type":"ContainerStarted","Data":"1650a03b0cc3f10227279720fad32aa12af011494cd3b71a7c7ca34811236aa4"} Nov 02 15:37:10.787627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:10.787528 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" podStartSLOduration=1.7874978559999999 podStartE2EDuration="1.787497856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:10.78325858 +0000 UTC m=+2384.947811119" watchObservedRunningTime="2026-11-02 15:37:10.787497856 +0000 UTC m=+2384.952050396" Nov 02 15:37:11.682387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:11.682350 5091 generic.go:356] "Generic (PLEG): container finished" podID="dd7e188c-0a25-49be-a5e3-fb705bc86a91" containerID="24675509e1760a3b06621e81d0581b815a32f0838eb8ef3620b65bfc4c10c6af" exitCode=0 Nov 02 15:37:11.682820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:11.682434 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" event={"ID":"dd7e188c-0a25-49be-a5e3-fb705bc86a91","Type":"ContainerDied","Data":"24675509e1760a3b06621e81d0581b815a32f0838eb8ef3620b65bfc4c10c6af"} Nov 02 15:37:12.864229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.864202 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:12.937423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.937387 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca\") pod \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " Nov 02 15:37:12.937604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.937452 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca\") pod \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " Nov 02 15:37:12.937670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.937633 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r\" (UniqueName: \"kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r\") pod \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\" (UID: \"dd7e188c-0a25-49be-a5e3-fb705bc86a91\") " Nov 02 15:37:12.937881 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.937853 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca" pod "dd7e188c-0a25-49be-a5e3-fb705bc86a91" (UID: "dd7e188c-0a25-49be-a5e3-fb705bc86a91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:12.938212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.938183 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:12.939550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.939511 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca" pod "dd7e188c-0a25-49be-a5e3-fb705bc86a91" (UID: "dd7e188c-0a25-49be-a5e3-fb705bc86a91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:12.940168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:12.940123 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r" pod "dd7e188c-0a25-49be-a5e3-fb705bc86a91" (UID: "dd7e188c-0a25-49be-a5e3-fb705bc86a91"). InnerVolumeSpecName "kube-api-access-ldl2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:13.053837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:13.053795 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldl2r\" (UniqueName: \"kubernetes.io/projected/dd7e188c-0a25-49be-a5e3-fb705bc86a91-kube-api-access-ldl2r\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:13.053837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:13.053836 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7e188c-0a25-49be-a5e3-fb705bc86a91-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:13.706370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:13.706338 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" Nov 02 15:37:13.706605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:13.706377 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj" event={"ID":"dd7e188c-0a25-49be-a5e3-fb705bc86a91","Type":"ContainerDied","Data":"1650a03b0cc3f10227279720fad32aa12af011494cd3b71a7c7ca34811236aa4"} Nov 02 15:37:13.706605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:13.706460 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1650a03b0cc3f10227279720fad32aa12af011494cd3b71a7c7ca34811236aa4" Nov 02 15:37:15.053130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:15.053098 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn"] Nov 02 15:37:15.097601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:15.097571 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdf9737e-rrlqn"] Nov 02 15:37:15.965060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:15.965021 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="764b0102-b952-40b0-998a-d918ea06512d" path="/var/lib/kubelet/pods/764b0102-b952-40b0-998a-d918ea06512d/volumes" Nov 02 15:37:24.076690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.076655 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4"] Nov 02 15:37:24.077311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.077288 5091 cpu_manager.go:414] "containerMap: removing container" podUID="dd7e188c-0a25-49be-a5e3-fb705bc86a91" containerName="revoke-keys" Nov 02 15:37:24.077355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.077311 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7e188c-0a25-49be-a5e3-fb705bc86a91" containerName="revoke-keys" Nov 02 15:37:24.077410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.077401 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7e188c-0a25-49be-a5e3-fb705bc86a91" containerName="revoke-keys" Nov 02 15:37:24.080783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.080759 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.137401 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.137369 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4"] Nov 02 15:37:24.177738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.177703 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:37:24.177738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.177724 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:37:24.264436 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.264403 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.264644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.264444 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94h6v\" (UniqueName: \"kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.264644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.264628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.365880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.365853 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.366027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.365892 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94h6v\" (UniqueName: \"kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.366027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.365944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.366718 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.366692 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.368329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.368309 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.424354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.424317 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94h6v\" (UniqueName: \"kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v\") pod \"maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:24.694819 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:24.694741 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:25.073086 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:25.073006 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4"] Nov 02 15:37:25.077293 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:37:25.077263 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521757a0_e831_4c6a_b5a1_e4104aa0f47a.slice/crio-e72f7b422f064c511249b68713ae1dea547173e12f010050b2b01912d9a1375b WatchSource:0}: Error finding container e72f7b422f064c511249b68713ae1dea547173e12f010050b2b01912d9a1375b: Status 404 returned error can't find the container with id e72f7b422f064c511249b68713ae1dea547173e12f010050b2b01912d9a1375b Nov 02 15:37:25.805100 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:25.805054 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" event={"ID":"521757a0-e831-4c6a-b5a1-e4104aa0f47a","Type":"ContainerStarted","Data":"2230bcd6d1a54be791ea1401a71c13e8b093c508c0a1e7770b6c5f92a5d036b8"} Nov 02 15:37:25.805281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:25.805129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" event={"ID":"521757a0-e831-4c6a-b5a1-e4104aa0f47a","Type":"ContainerStarted","Data":"e72f7b422f064c511249b68713ae1dea547173e12f010050b2b01912d9a1375b"} Nov 02 15:37:25.920171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:25.920116 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" podStartSLOduration=1.920098724 podStartE2EDuration="1.920098724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:37:25.915245655 +0000 UTC m=+2400.079798197" watchObservedRunningTime="2026-11-02 15:37:25.920098724 +0000 UTC m=+2400.084651266" Nov 02 15:37:26.813021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:26.812986 5091 generic.go:356] "Generic (PLEG): container finished" podID="521757a0-e831-4c6a-b5a1-e4104aa0f47a" containerID="2230bcd6d1a54be791ea1401a71c13e8b093c508c0a1e7770b6c5f92a5d036b8" exitCode=0 Nov 02 15:37:26.813438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:26.813071 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" event={"ID":"521757a0-e831-4c6a-b5a1-e4104aa0f47a","Type":"ContainerDied","Data":"2230bcd6d1a54be791ea1401a71c13e8b093c508c0a1e7770b6c5f92a5d036b8"} Nov 02 15:37:27.964153 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:37:27.964070 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:37:28.045055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.045031 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:28.125787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.125754 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca\") pod \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " Nov 02 15:37:28.125965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.125818 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v\" (UniqueName: \"kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v\") pod \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " Nov 02 15:37:28.126184 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.126155 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca\") pod \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\" (UID: \"521757a0-e831-4c6a-b5a1-e4104aa0f47a\") " Nov 02 15:37:28.126645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.126581 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca" pod "521757a0-e831-4c6a-b5a1-e4104aa0f47a" (UID: "521757a0-e831-4c6a-b5a1-e4104aa0f47a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:28.126994 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.126969 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:28.128597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.128572 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v" pod "521757a0-e831-4c6a-b5a1-e4104aa0f47a" (UID: "521757a0-e831-4c6a-b5a1-e4104aa0f47a"). InnerVolumeSpecName "kube-api-access-94h6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:28.128715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.128687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca" pod "521757a0-e831-4c6a-b5a1-e4104aa0f47a" (UID: "521757a0-e831-4c6a-b5a1-e4104aa0f47a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:28.228563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.228498 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/521757a0-e831-4c6a-b5a1-e4104aa0f47a-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:28.228563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.228559 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94h6v\" (UniqueName: \"kubernetes.io/projected/521757a0-e831-4c6a-b5a1-e4104aa0f47a-kube-api-access-94h6v\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:28.714710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.714672 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:37:28.732659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.732637 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:37:28.839054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.839025 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" Nov 02 15:37:28.839054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.839040 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4" event={"ID":"521757a0-e831-4c6a-b5a1-e4104aa0f47a","Type":"ContainerDied","Data":"e72f7b422f064c511249b68713ae1dea547173e12f010050b2b01912d9a1375b"} Nov 02 15:37:28.839329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:28.839094 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e72f7b422f064c511249b68713ae1dea547173e12f010050b2b01912d9a1375b" Nov 02 15:37:38.060952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.060915 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l"] Nov 02 15:37:38.061470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.061158 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" podUID="298339d2-4e94-4908-bee4-a9cae13936c8" containerName="maas-api" containerID="cri-o://6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22" gracePeriod=30 Nov 02 15:37:38.425774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.425749 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:37:38.616188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.613807 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\") pod \"298339d2-4e94-4908-bee4-a9cae13936c8\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " Nov 02 15:37:38.616188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.614089 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855\" (UniqueName: \"kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855\") pod \"298339d2-4e94-4908-bee4-a9cae13936c8\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " Nov 02 15:37:38.616188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.614630 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca\") pod \"298339d2-4e94-4908-bee4-a9cae13936c8\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " Nov 02 15:37:38.616188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.614688 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs\") pod \"298339d2-4e94-4908-bee4-a9cae13936c8\" (UID: \"298339d2-4e94-4908-bee4-a9cae13936c8\") " Nov 02 15:37:38.616188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.616143 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca" pod "298339d2-4e94-4908-bee4-a9cae13936c8" (UID: "298339d2-4e94-4908-bee4-a9cae13936c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:38.624784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.621007 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls" pod "298339d2-4e94-4908-bee4-a9cae13936c8" (UID: "298339d2-4e94-4908-bee4-a9cae13936c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:38.624784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.624647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs" pod "298339d2-4e94-4908-bee4-a9cae13936c8" (UID: "298339d2-4e94-4908-bee4-a9cae13936c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:37:38.625406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.625377 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855" pod "298339d2-4e94-4908-bee4-a9cae13936c8" (UID: "298339d2-4e94-4908-bee4-a9cae13936c8"). InnerVolumeSpecName "kube-api-access-vv855". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:38.716952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.716861 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vv855\" (UniqueName: \"kubernetes.io/projected/298339d2-4e94-4908-bee4-a9cae13936c8-kube-api-access-vv855\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:38.716952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.716901 5091 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/298339d2-4e94-4908-bee4-a9cae13936c8-openshift-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:38.716952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.716918 5091 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-metrics-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:38.716952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.716934 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/298339d2-4e94-4908-bee4-a9cae13936c8-maas-api-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:38.915655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.915620 5091 generic.go:356] "Generic (PLEG): container finished" podID="298339d2-4e94-4908-bee4-a9cae13936c8" containerID="6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22" exitCode=0 Nov 02 15:37:38.915826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.915667 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" Nov 02 15:37:38.915826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.915682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" event={"ID":"298339d2-4e94-4908-bee4-a9cae13936c8","Type":"ContainerDied","Data":"6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22"} Nov 02 15:37:38.915826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.915732 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l" event={"ID":"298339d2-4e94-4908-bee4-a9cae13936c8","Type":"ContainerDied","Data":"cc933db19513d4a281e7df31f09290b47cce974b519a05a070a9069cc7b6227a"} Nov 02 15:37:38.915826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.915751 5091 scope.go:122] "RemoveContainer" containerID="6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22" Nov 02 15:37:38.929677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.929651 5091 scope.go:122] "RemoveContainer" containerID="6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22" Nov 02 15:37:38.929989 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:37:38.929964 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22\": container with ID starting with 6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22 not found: ID does not exist" containerID="6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22" Nov 02 15:37:38.930074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:38.930003 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22"} err="failed to get container status \"6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22\": rpc error: code = NotFound desc = could not find container \"6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22\": container with ID starting with 6e0ccd796457e32d9a72517700d842c8430e36e5ea2aa84a552d809ac6393c22 not found: ID does not exist" Nov 02 15:37:39.145542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:39.139460 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l"] Nov 02 15:37:39.169539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:39.165337 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-adf096-c65b9b6d8-6ww9l"] Nov 02 15:37:39.962246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:39.962203 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="298339d2-4e94-4908-bee4-a9cae13936c8" path="/var/lib/kubelet/pods/298339d2-4e94-4908-bee4-a9cae13936c8/volumes" Nov 02 15:37:40.420943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:40.420906 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw"] Nov 02 15:37:40.424335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:40.424300 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-adf096-rj2mw"] Nov 02 15:37:41.963437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:41.963392 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43daa66-75ba-4e58-87c0-a8caa6c909af" path="/var/lib/kubelet/pods/a43daa66-75ba-4e58-87c0-a8caa6c909af/volumes" Nov 02 15:37:47.420941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:47.420907 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp"] Nov 02 15:37:47.421612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:47.421132 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" containerID="cri-o://bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929" gracePeriod=30 Nov 02 15:37:50.661082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:50.661046 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:37:50.661473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:50.661108 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:37:52.749664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.749635 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:37:52.851731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.851701 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.851900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.851772 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.851900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.851790 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.851957 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852095 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852150 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:37:52.852334 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852269 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852356 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:52.852470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852450 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852552 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852586 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:52.852624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852593 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token\") pod \"e88f8840-f89d-4ec4-ae7c-a688205a452e\" (UID: \"e88f8840-f89d-4ec4-ae7c-a688205a452e\") " Nov 02 15:37:52.852794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.852694 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:52.853369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.853343 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.853473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.853373 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.853473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.853388 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.853473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.853402 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88f8840-f89d-4ec4-ae7c-a688205a452e-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.853704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.853612 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:52.854234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.854184 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "kube-api-access-knsln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:52.854234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.854194 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:37:52.854355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.854270 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:37:52.854867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.854844 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token" pod "e88f8840-f89d-4ec4-ae7c-a688205a452e" (UID: "e88f8840-f89d-4ec4-ae7c-a688205a452e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:37:52.955038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.954904 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.955038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.954945 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.955038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.954959 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.955038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.954974 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knsln\" (UniqueName: \"kubernetes.io/projected/e88f8840-f89d-4ec4-ae7c-a688205a452e-kube-api-access-knsln\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:52.955038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:52.954988 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88f8840-f89d-4ec4-ae7c-a688205a452e-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:37:53.070388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.070300 5091 generic.go:356] "Generic (PLEG): container finished" podID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerID="bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929" exitCode=0 Nov 02 15:37:53.070388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.070361 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" Nov 02 15:37:53.070610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.070380 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" event={"ID":"e88f8840-f89d-4ec4-ae7c-a688205a452e","Type":"ContainerDied","Data":"bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929"} Nov 02 15:37:53.070610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.070442 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp" event={"ID":"e88f8840-f89d-4ec4-ae7c-a688205a452e","Type":"ContainerDied","Data":"e4aa112cafc08ad001450631ca086316ba24b9d00b6711694d5d80398a7ff67a"} Nov 02 15:37:53.070610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.070467 5091 scope.go:122] "RemoveContainer" containerID="bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929" Nov 02 15:37:53.089607 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.089573 5091 scope.go:122] "RemoveContainer" containerID="ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc" Nov 02 15:37:53.101057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.101026 5091 scope.go:122] "RemoveContainer" containerID="bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929" Nov 02 15:37:53.101332 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:37:53.101309 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929\": container with ID starting with bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929 not found: ID does not exist" containerID="bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929" Nov 02 15:37:53.101416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.101345 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929"} err="failed to get container status \"bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929\": rpc error: code = NotFound desc = could not find container \"bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929\": container with ID starting with bbe0aea35e8f48a8893cddda5fe8cd780cf18ae23efb968844aaa63e4d977929 not found: ID does not exist" Nov 02 15:37:53.101416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.101370 5091 scope.go:122] "RemoveContainer" containerID="ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc" Nov 02 15:37:53.101660 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:37:53.101640 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc\": container with ID starting with ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc not found: ID does not exist" containerID="ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc" Nov 02 15:37:53.101745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.101666 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc"} err="failed to get container status \"ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc\": rpc error: code = NotFound desc = could not find container \"ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc\": container with ID starting with ed4d05c09aad1b40f56d44c896554d1dac3dbbda0aaaf2f0cf802fc58a0368dc not found: ID does not exist" Nov 02 15:37:53.229595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.229563 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp"] Nov 02 15:37:53.235968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.235941 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-adf096-openshift-default-67bdb4dd9f-8h4vp"] Nov 02 15:37:53.961414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:53.961381 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" path="/var/lib/kubelet/pods/e88f8840-f89d-4ec4-ae7c-a688205a452e/volumes" Nov 02 15:37:57.936400 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:37:57.936364 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:37:58.888261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:58.888225 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r"] Nov 02 15:37:58.934701 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:58.934668 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-5afebc-qhk5r"] Nov 02 15:37:59.962703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:37:59.962668 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1f4705-17ed-4447-8c31-3baedc0c4f07" path="/var/lib/kubelet/pods/1f1f4705-17ed-4447-8c31-3baedc0c4f07/volumes" Nov 02 15:38:06.320971 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:06.320933 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z"] Nov 02 15:38:06.321491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:06.321195 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" containerID="cri-o://fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0" gracePeriod=30 Nov 02 15:38:08.057684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.057647 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4"] Nov 02 15:38:08.058659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058608 5091 cpu_manager.go:414] "containerMap: removing container" podUID="521757a0-e831-4c6a-b5a1-e4104aa0f47a" containerName="revoke-keys" Nov 02 15:38:08.058659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058632 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521757a0-e831-4c6a-b5a1-e4104aa0f47a" containerName="revoke-keys" Nov 02 15:38:08.058813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058683 5091 cpu_manager.go:414] "containerMap: removing container" podUID="298339d2-4e94-4908-bee4-a9cae13936c8" containerName="maas-api" Nov 02 15:38:08.058813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058692 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="298339d2-4e94-4908-bee4-a9cae13936c8" containerName="maas-api" Nov 02 15:38:08.058813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058703 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" Nov 02 15:38:08.058813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058711 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" Nov 02 15:38:08.058813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058738 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" Nov 02 15:38:08.058813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058746 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" Nov 02 15:38:08.059010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058909 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" Nov 02 15:38:08.059010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058930 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="521757a0-e831-4c6a-b5a1-e4104aa0f47a" containerName="revoke-keys" Nov 02 15:38:08.059010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.058947 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="298339d2-4e94-4908-bee4-a9cae13936c8" containerName="maas-api" Nov 02 15:38:08.064706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.064685 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.116834 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.116769 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-eb6bd4-openshift-default-dockercfg-tpbn6\"" Nov 02 15:38:08.160728 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.160679 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.161049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.161028 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.161246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.161214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.161395 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.161378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.179876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.179678 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.179876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.179733 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5qp\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.179876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.179869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.180149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.179970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.180149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.179998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.180262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.178119 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4"] Nov 02 15:38:08.281066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281032 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281070 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281206 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281239 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281280 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281333 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5qp\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.281703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.281419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.282113 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.282087 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.282211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.282154 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.282264 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.282234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.282538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.282495 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.282657 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.282509 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.284088 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.284066 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.284582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.284557 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.379822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.376595 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5qp\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.403656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.403589 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token\") pod \"e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:08.696584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:08.689385 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:09.237081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:09.236826 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4"] Nov 02 15:38:09.243499 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:38:09.243467 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dc0efb_2f20_413e_9d8b_b26e02769233.slice/crio-73c1ebdc1b98bd6e78657b69cef417bea632fd18d8f0da966d6c62c65151b077 WatchSource:0}: Error finding container 73c1ebdc1b98bd6e78657b69cef417bea632fd18d8f0da966d6c62c65151b077: Status 404 returned error can't find the container with id 73c1ebdc1b98bd6e78657b69cef417bea632fd18d8f0da966d6c62c65151b077 Nov 02 15:38:09.246972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:09.246928 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:09.247068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:09.247015 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:09.247068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:09.247058 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:38:09.363270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:09.363187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" event={"ID":"36dc0efb-2f20-413e-9d8b-b26e02769233","Type":"ContainerStarted","Data":"73c1ebdc1b98bd6e78657b69cef417bea632fd18d8f0da966d6c62c65151b077"} Nov 02 15:38:10.373305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:10.373274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" event={"ID":"36dc0efb-2f20-413e-9d8b-b26e02769233","Type":"ContainerStarted","Data":"3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f"} Nov 02 15:38:10.547377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:10.547317 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podStartSLOduration=2.547298821 podStartE2EDuration="2.547298821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:10.542467445 +0000 UTC m=+2444.707019987" watchObservedRunningTime="2026-11-02 15:38:10.547298821 +0000 UTC m=+2444.711851364" Nov 02 15:38:10.607886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:10.607851 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn"] Nov 02 15:38:10.611820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:10.611796 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5aeac88e-xprxn"] Nov 02 15:38:10.700495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:10.700306 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:10.709409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:10.709379 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:11.386534 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.385902 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:11.388595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.388567 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:38:11.611713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.611641 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:38:11.759338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759308 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759348 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759429 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759468 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759511 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759663 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.759891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759734 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759822 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.759891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.759849 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy\") pod \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\" (UID: \"b61c5e58-10ce-4239-8f92-70f81bbcd7c6\") " Nov 02 15:38:11.760267 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760239 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.760267 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760254 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.760420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760317 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.760611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760588 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.760705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760616 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.760705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760631 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.760705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760646 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.760906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.760602 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:11.762149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.762122 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "kube-api-access-mjc47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.762753 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.762730 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:11.762842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.762770 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:11.762842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.762804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token" pod "b61c5e58-10ce-4239-8f92-70f81bbcd7c6" (UID: "b61c5e58-10ce-4239-8f92-70f81bbcd7c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:11.861929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.861854 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.861929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.861887 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjc47\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-kube-api-access-mjc47\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.861929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.861902 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.861929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.861913 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.861929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.861924 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b61c5e58-10ce-4239-8f92-70f81bbcd7c6-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:11.962083 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:11.962050 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c069f5f-a487-4791-b786-de9d0e7cf520" path="/var/lib/kubelet/pods/8c069f5f-a487-4791-b786-de9d0e7cf520/volumes" Nov 02 15:38:12.393961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.393930 5091 generic.go:356] "Generic (PLEG): container finished" podID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerID="fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0" exitCode=0 Nov 02 15:38:12.394406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.393975 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" Nov 02 15:38:12.394406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.394003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" event={"ID":"b61c5e58-10ce-4239-8f92-70f81bbcd7c6","Type":"ContainerDied","Data":"fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0"} Nov 02 15:38:12.394406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.394043 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z" event={"ID":"b61c5e58-10ce-4239-8f92-70f81bbcd7c6","Type":"ContainerDied","Data":"3a8aae516b055a07c844b854cd1de5913806c9d241dd7bfbf82250f6be703e06"} Nov 02 15:38:12.394406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.394066 5091 scope.go:122] "RemoveContainer" containerID="fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0" Nov 02 15:38:12.405170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.405147 5091 scope.go:122] "RemoveContainer" containerID="fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0" Nov 02 15:38:12.405505 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:38:12.405487 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0\": container with ID starting with fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0 not found: ID does not exist" containerID="fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0" Nov 02 15:38:12.405625 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.405510 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0"} err="failed to get container status \"fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0\": rpc error: code = NotFound desc = could not find container \"fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0\": container with ID starting with fb4ed9a5a027d85fde0fe0c189b3ae8d22c008a572369fe5ba06769aa9580aa0 not found: ID does not exist" Nov 02 15:38:12.553039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.553006 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z"] Nov 02 15:38:12.599542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:12.599484 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5afebc-openshift-default-6b4f54845-4r42z"] Nov 02 15:38:13.980125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:13.980083 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" path="/var/lib/kubelet/pods/b61c5e58-10ce-4239-8f92-70f81bbcd7c6/volumes" Nov 02 15:38:22.211735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.211694 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv"] Nov 02 15:38:22.212626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.212501 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" Nov 02 15:38:22.212626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.212543 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" Nov 02 15:38:22.213692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.212701 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88f8840-f89d-4ec4-ae7c-a688205a452e" containerName="istio-proxy" Nov 02 15:38:22.213692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.212712 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b61c5e58-10ce-4239-8f92-70f81bbcd7c6" containerName="istio-proxy" Nov 02 15:38:22.216207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.216190 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.253828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.253800 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv"] Nov 02 15:38:22.294296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.294274 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f6b245\"" Nov 02 15:38:22.294457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.294353 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f6b245-dockercfg-4xmcq\"" Nov 02 15:38:22.339568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.339496 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pj4q\" (UniqueName: \"kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q\") pod \"payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.339734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.339593 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.440670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.440621 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pj4q\" (UniqueName: \"kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q\") pod \"payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.440881 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.440738 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.441637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.441611 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.500481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.500419 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pj4q\" (UniqueName: \"kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q\") pod \"payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.534025 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.533982 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:22.925597 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:38:22.925566 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27b3e27_f31e_40e8_991f_814e9922e08e.slice/crio-c4d3bf090df03a9232e5c78cf0c07362b6a3c0615bf6a7856735f74f9f1d3a2b WatchSource:0}: Error finding container c4d3bf090df03a9232e5c78cf0c07362b6a3c0615bf6a7856735f74f9f1d3a2b: Status 404 returned error can't find the container with id c4d3bf090df03a9232e5c78cf0c07362b6a3c0615bf6a7856735f74f9f1d3a2b Nov 02 15:38:22.961490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:22.961421 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv"] Nov 02 15:38:23.489556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:23.489491 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" event={"ID":"a27b3e27-f31e-40e8-991f-814e9922e08e","Type":"ContainerStarted","Data":"b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f"} Nov 02 15:38:23.489968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:23.489586 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:23.489968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:23.489605 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" event={"ID":"a27b3e27-f31e-40e8-991f-814e9922e08e","Type":"ContainerStarted","Data":"c4d3bf090df03a9232e5c78cf0c07362b6a3c0615bf6a7856735f74f9f1d3a2b"} Nov 02 15:38:23.611662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:23.611612 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" podStartSLOduration=1.611596607 podStartE2EDuration="1.611596607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:23.607623649 +0000 UTC m=+2457.772176190" watchObservedRunningTime="2026-11-02 15:38:23.611596607 +0000 UTC m=+2457.776149147" Nov 02 15:38:27.935084 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:38:27.935046 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:38:29.531929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:29.531894 5091 scope.go:122] "RemoveContainer" containerID="e04e381279841b761111dcfc9a896f859c44eb0dbaee007ddf40abfe6539ba8c" Nov 02 15:38:29.542756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:29.542735 5091 scope.go:122] "RemoveContainer" containerID="12cc24b7bed01368359b70af7194f6411ea2d6bc68690edd093dee2056f465c7" Nov 02 15:38:29.556439 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:29.556415 5091 scope.go:122] "RemoveContainer" containerID="cea0652e0eeb1a90adc8aeb1d8c96053e306abbea4cb99b82bd7bb0d220ed5db" Nov 02 15:38:29.566508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:29.566485 5091 scope.go:122] "RemoveContainer" containerID="28072a4382cbba37459718831395ec2564dee6ed37fe35bdbb6693926f52f633" Nov 02 15:38:31.053468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:31.053434 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r"] Nov 02 15:38:31.099815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:31.099785 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862da326-pk26r"] Nov 02 15:38:31.961386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:31.961343 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12cd230-cfd0-4d6b-b8f3-2fe913b451a4" path="/var/lib/kubelet/pods/f12cd230-cfd0-4d6b-b8f3-2fe913b451a4/volumes" Nov 02 15:38:34.499471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:34.499439 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:38:34.863200 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:34.863110 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj"] Nov 02 15:38:34.867717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:34.867686 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa5832-tfkmj"] Nov 02 15:38:35.961429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:35.961397 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7e188c-0a25-49be-a5e3-fb705bc86a91" path="/var/lib/kubelet/pods/dd7e188c-0a25-49be-a5e3-fb705bc86a91/volumes" Nov 02 15:38:41.636016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:41.635932 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4"] Nov 02 15:38:41.685478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:41.685434 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-822f77-qd9l4"] Nov 02 15:38:42.015589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:42.010825 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521757a0-e831-4c6a-b5a1-e4104aa0f47a" path="/var/lib/kubelet/pods/521757a0-e831-4c6a-b5a1-e4104aa0f47a/volumes" Nov 02 15:38:44.210861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.210825 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz"] Nov 02 15:38:44.216932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.216913 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.268018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.267985 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:38:44.268169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.268090 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:38:44.273241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.273215 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz"] Nov 02 15:38:44.340775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.340745 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.340922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.340845 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.341001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.340985 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv7tv\" (UniqueName: \"kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.441982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.441951 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv7tv\" (UniqueName: \"kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.442176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.442098 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.442176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.442169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.443078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.443049 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.444815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.444789 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.500962 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.500899 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv7tv\" (UniqueName: \"kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv\") pod \"maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.530986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.530961 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:44.919875 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:44.919838 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz"] Nov 02 15:38:44.923826 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:38:44.923790 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c6cdfb_e580_44c5_85ff_b50ab85454b0.slice/crio-35f2086b7d22595292869333303d9dcbea0b3d98fae7c015aeafe841247700bf WatchSource:0}: Error finding container 35f2086b7d22595292869333303d9dcbea0b3d98fae7c015aeafe841247700bf: Status 404 returned error can't find the container with id 35f2086b7d22595292869333303d9dcbea0b3d98fae7c015aeafe841247700bf Nov 02 15:38:45.675114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:45.675075 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" event={"ID":"d2c6cdfb-e580-44c5-85ff-b50ab85454b0","Type":"ContainerStarted","Data":"7520ef3c46013b8beee0f46dccd9d91270473b90fbab015ca40cd6b21b892b80"} Nov 02 15:38:45.675670 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:45.675132 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" event={"ID":"d2c6cdfb-e580-44c5-85ff-b50ab85454b0","Type":"ContainerStarted","Data":"35f2086b7d22595292869333303d9dcbea0b3d98fae7c015aeafe841247700bf"} Nov 02 15:38:45.797850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:45.797799 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" podStartSLOduration=1.797783846 podStartE2EDuration="1.797783846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:38:45.796062369 +0000 UTC m=+2479.960614909" watchObservedRunningTime="2026-11-02 15:38:45.797783846 +0000 UTC m=+2479.962336385" Nov 02 15:38:46.685012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:46.684981 5091 generic.go:356] "Generic (PLEG): container finished" podID="d2c6cdfb-e580-44c5-85ff-b50ab85454b0" containerID="7520ef3c46013b8beee0f46dccd9d91270473b90fbab015ca40cd6b21b892b80" exitCode=0 Nov 02 15:38:46.685441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:46.685067 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" event={"ID":"d2c6cdfb-e580-44c5-85ff-b50ab85454b0","Type":"ContainerDied","Data":"7520ef3c46013b8beee0f46dccd9d91270473b90fbab015ca40cd6b21b892b80"} Nov 02 15:38:47.873370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.873342 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:47.901779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.901755 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca\") pod \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " Nov 02 15:38:47.901902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.901823 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv\" (UniqueName: \"kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv\") pod \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " Nov 02 15:38:47.901902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.901861 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca\") pod \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\" (UID: \"d2c6cdfb-e580-44c5-85ff-b50ab85454b0\") " Nov 02 15:38:47.902283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.902255 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca" pod "d2c6cdfb-e580-44c5-85ff-b50ab85454b0" (UID: "d2c6cdfb-e580-44c5-85ff-b50ab85454b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:47.902565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.902545 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:47.903890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.903862 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca" pod "d2c6cdfb-e580-44c5-85ff-b50ab85454b0" (UID: "d2c6cdfb-e580-44c5-85ff-b50ab85454b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:47.903994 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:47.903889 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv" pod "d2c6cdfb-e580-44c5-85ff-b50ab85454b0" (UID: "d2c6cdfb-e580-44c5-85ff-b50ab85454b0"). InnerVolumeSpecName "kube-api-access-hv7tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:48.004841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.004801 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv7tv\" (UniqueName: \"kubernetes.io/projected/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-kube-api-access-hv7tv\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:48.004841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.004841 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2c6cdfb-e580-44c5-85ff-b50ab85454b0-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:48.678023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.677992 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs"] Nov 02 15:38:48.688643 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.688607 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq"] Nov 02 15:38:48.710328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.710306 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" Nov 02 15:38:48.710475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.710349 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz" event={"ID":"d2c6cdfb-e580-44c5-85ff-b50ab85454b0","Type":"ContainerDied","Data":"35f2086b7d22595292869333303d9dcbea0b3d98fae7c015aeafe841247700bf"} Nov 02 15:38:48.710475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:48.710398 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f2086b7d22595292869333303d9dcbea0b3d98fae7c015aeafe841247700bf" Nov 02 15:38:49.203912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:49.203874 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd"] Nov 02 15:38:49.204399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:49.204079 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" containerID="cri-o://37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b" gracePeriod=30 Nov 02 15:38:51.606064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.606029 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc"] Nov 02 15:38:51.606914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.606891 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d2c6cdfb-e580-44c5-85ff-b50ab85454b0" containerName="revoke-keys" Nov 02 15:38:51.606914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.606913 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c6cdfb-e580-44c5-85ff-b50ab85454b0" containerName="revoke-keys" Nov 02 15:38:51.607128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.607113 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c6cdfb-e580-44c5-85ff-b50ab85454b0" containerName="revoke-keys" Nov 02 15:38:51.610981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.610958 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.661571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.661504 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:38:51.661721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.661615 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:38:51.666598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.666567 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc"] Nov 02 15:38:51.712419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.712385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.712619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.712458 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p27c\" (UniqueName: \"kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.712743 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.712713 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.814547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.814471 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.814747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.814610 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.814747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.814675 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p27c\" (UniqueName: \"kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.815243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.815216 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.817012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.816990 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.874465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.874414 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p27c\" (UniqueName: \"kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c\") pod \"maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:51.925771 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:51.925744 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:52.307220 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:52.307145 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc"] Nov 02 15:38:52.311256 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:38:52.311227 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52a4144_a48a_4e12_88c6_978fc38c14cc.slice/crio-eb1b5fc6639c2b7c2de8070966bfbb4bb4e5e13feabcaf8b8e846263dbb2baae WatchSource:0}: Error finding container eb1b5fc6639c2b7c2de8070966bfbb4bb4e5e13feabcaf8b8e846263dbb2baae: Status 404 returned error can't find the container with id eb1b5fc6639c2b7c2de8070966bfbb4bb4e5e13feabcaf8b8e846263dbb2baae Nov 02 15:38:52.746038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:52.745998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" event={"ID":"b52a4144-a48a-4e12-88c6-978fc38c14cc","Type":"ContainerStarted","Data":"eb1b5fc6639c2b7c2de8070966bfbb4bb4e5e13feabcaf8b8e846263dbb2baae"} Nov 02 15:38:53.755843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:53.755766 5091 generic.go:356] "Generic (PLEG): container finished" podID="b52a4144-a48a-4e12-88c6-978fc38c14cc" containerID="486e66036245ecf21bad1622806271690f3c737efe3e9354edc5efba1a357150" exitCode=0 Nov 02 15:38:53.756233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:53.755843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" event={"ID":"b52a4144-a48a-4e12-88c6-978fc38c14cc","Type":"ContainerDied","Data":"486e66036245ecf21bad1622806271690f3c737efe3e9354edc5efba1a357150"} Nov 02 15:38:54.489762 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.489674 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:38:54.560944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.560907 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561034 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561100 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561115 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:54.561239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561145 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561182 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561211 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561266 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561466 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy\") pod \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\" (UID: \"e8f984c6-f521-4ebd-8b59-602ebcbc3abd\") " Nov 02 15:38:54.561621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561480 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:54.561621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561495 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:54.561621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561504 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:54.561621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.561584 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:54.562135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.562110 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.562233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.562140 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.562233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.562152 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.562233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.562161 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.562233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.562175 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.563730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.563707 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:38:54.563829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.563728 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "kube-api-access-dgft7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:54.564066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.564045 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:38:54.564133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.564105 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token" pod "e8f984c6-f521-4ebd-8b59-602ebcbc3abd" (UID: "e8f984c6-f521-4ebd-8b59-602ebcbc3abd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:54.663710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.663674 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgft7\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-kube-api-access-dgft7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.663710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.663708 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.663906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.663722 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.663906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.663738 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f984c6-f521-4ebd-8b59-602ebcbc3abd-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.765321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.765236 5091 generic.go:356] "Generic (PLEG): container finished" podID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerID="37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b" exitCode=0 Nov 02 15:38:54.765321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.765306 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" Nov 02 15:38:54.765884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.765308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" event={"ID":"e8f984c6-f521-4ebd-8b59-602ebcbc3abd","Type":"ContainerDied","Data":"37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b"} Nov 02 15:38:54.765884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.765380 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd" event={"ID":"e8f984c6-f521-4ebd-8b59-602ebcbc3abd","Type":"ContainerDied","Data":"9f2048856f0fe26d6bd0244537e7ae20e83f97fb1a1d53c635a98edf7b47e05a"} Nov 02 15:38:54.765884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.765413 5091 scope.go:122] "RemoveContainer" containerID="37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b" Nov 02 15:38:54.780168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.780147 5091 scope.go:122] "RemoveContainer" containerID="1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970" Nov 02 15:38:54.799545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.796032 5091 scope.go:122] "RemoveContainer" containerID="37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b" Nov 02 15:38:54.799545 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:38:54.798806 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b\": container with ID starting with 37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b not found: ID does not exist" containerID="37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b" Nov 02 15:38:54.799545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.798845 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b"} err="failed to get container status \"37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b\": rpc error: code = NotFound desc = could not find container \"37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b\": container with ID starting with 37dd434f819334d57bcf788386344b12d05ebf17cfcaa6f58ba8cc0f02814a6b not found: ID does not exist" Nov 02 15:38:54.799545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.798873 5091 scope.go:122] "RemoveContainer" containerID="1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970" Nov 02 15:38:54.802038 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:38:54.800311 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970\": container with ID starting with 1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970 not found: ID does not exist" containerID="1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970" Nov 02 15:38:54.802038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.800341 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970"} err="failed to get container status \"1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970\": rpc error: code = NotFound desc = could not find container \"1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970\": container with ID starting with 1c1667f04ea01c43f81cc149bc51b16243066638e93efe5bf97fbe759ddbe970 not found: ID does not exist" Nov 02 15:38:54.928013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.927981 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd"] Nov 02 15:38:54.931563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.931537 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:54.936102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.936073 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-822f77-openshift-default-79db8bfdf9-xl9rd"] Nov 02 15:38:54.966601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.966574 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c\" (UniqueName: \"kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c\") pod \"b52a4144-a48a-4e12-88c6-978fc38c14cc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " Nov 02 15:38:54.966761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.966741 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca\") pod \"b52a4144-a48a-4e12-88c6-978fc38c14cc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " Nov 02 15:38:54.966823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.966770 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca\") pod \"b52a4144-a48a-4e12-88c6-978fc38c14cc\" (UID: \"b52a4144-a48a-4e12-88c6-978fc38c14cc\") " Nov 02 15:38:54.967398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.967363 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca" pod "b52a4144-a48a-4e12-88c6-978fc38c14cc" (UID: "b52a4144-a48a-4e12-88c6-978fc38c14cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:38:54.972538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.969584 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:54.980026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.979992 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c" pod "b52a4144-a48a-4e12-88c6-978fc38c14cc" (UID: "b52a4144-a48a-4e12-88c6-978fc38c14cc"). InnerVolumeSpecName "kube-api-access-6p27c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:38:54.981950 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:54.981919 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca" pod "b52a4144-a48a-4e12-88c6-978fc38c14cc" (UID: "b52a4144-a48a-4e12-88c6-978fc38c14cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:38:55.070721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.070635 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a4144-a48a-4e12-88c6-978fc38c14cc-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:55.070721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.070672 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p27c\" (UniqueName: \"kubernetes.io/projected/b52a4144-a48a-4e12-88c6-978fc38c14cc-kube-api-access-6p27c\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:38:55.774895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.774860 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" Nov 02 15:38:55.775362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.774885 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc" event={"ID":"b52a4144-a48a-4e12-88c6-978fc38c14cc","Type":"ContainerDied","Data":"eb1b5fc6639c2b7c2de8070966bfbb4bb4e5e13feabcaf8b8e846263dbb2baae"} Nov 02 15:38:55.775362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.774976 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb1b5fc6639c2b7c2de8070966bfbb4bb4e5e13feabcaf8b8e846263dbb2baae" Nov 02 15:38:55.964382 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.964340 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" path="/var/lib/kubelet/pods/e8f984c6-f521-4ebd-8b59-602ebcbc3abd/volumes" Nov 02 15:38:55.987958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:38:55.987928 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq"] Nov 02 15:38:57.935850 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:38:57.935813 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:39:00.335007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:00.334971 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4"] Nov 02 15:39:00.335556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:00.335199 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" containerID="cri-o://57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b" gracePeriod=30 Nov 02 15:39:00.351340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:00.351307 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj"] Nov 02 15:39:00.351579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:00.351554 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" containerID="cri-o://fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63" gracePeriod=30 Nov 02 15:39:01.877902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:01.877868 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:01.878318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:01.877921 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:02.944724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.944688 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx"] Nov 02 15:39:02.945506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945483 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" Nov 02 15:39:02.945506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945509 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" Nov 02 15:39:02.945717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945590 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" Nov 02 15:39:02.945717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945600 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" Nov 02 15:39:02.945717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945621 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b52a4144-a48a-4e12-88c6-978fc38c14cc" containerName="revoke-keys" Nov 02 15:39:02.945717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945629 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52a4144-a48a-4e12-88c6-978fc38c14cc" containerName="revoke-keys" Nov 02 15:39:02.945912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945795 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" Nov 02 15:39:02.945912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.945819 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52a4144-a48a-4e12-88c6-978fc38c14cc" containerName="revoke-keys" Nov 02 15:39:02.950977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:02.950952 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.003120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.003091 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx"] Nov 02 15:39:03.006573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.006539 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.006709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.006627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff22t\" (UniqueName: \"kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.006842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.006824 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.048114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.048074 5091 patch_prober.go:29] interesting pod/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:03.048352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.048130 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:03.048464 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.048442 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:39:03.048595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.048573 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:39:03.052382 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.052356 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt"] Nov 02 15:39:03.053487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.053465 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f984c6-f521-4ebd-8b59-602ebcbc3abd" containerName="istio-proxy" Nov 02 15:39:03.057681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.057661 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.064572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.064546 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt"] Nov 02 15:39:03.105010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.104976 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6"] Nov 02 15:39:03.108358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.108329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff22t\" (UniqueName: \"kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.108476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.108451 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.108584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.108570 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.109358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.109337 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.110879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.110857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.166827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.166793 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff22t\" (UniqueName: \"kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t\") pod \"maas-api-revoke-sub-8b213b5a-kdmxx\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.205755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.205675 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw"] Nov 02 15:39:03.216692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.216660 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.216863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.216750 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.216938 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.216894 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xf92\" (UniqueName: \"kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.266818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.266786 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:03.318010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.317971 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.318225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.318052 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.318225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.318169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xf92\" (UniqueName: \"kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.318943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.318918 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.320778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.320752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.377310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.377275 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xf92\" (UniqueName: \"kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92\") pod \"maas-api-revoke-sub-b18037ab-n5swt\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.377579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.377553 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:03.652583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.650860 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx"] Nov 02 15:39:03.656476 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:39:03.656446 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c006b75_3951_4ccb_afe4_e685f98dda8e.slice/crio-0080de45316ef25637436e797d7ba02282a607a18410be34221d61957e32be76 WatchSource:0}: Error finding container 0080de45316ef25637436e797d7ba02282a607a18410be34221d61957e32be76: Status 404 returned error can't find the container with id 0080de45316ef25637436e797d7ba02282a607a18410be34221d61957e32be76 Nov 02 15:39:03.699627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.699547 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" containerID="cri-o://c807b53a2c7699021117f3f29bb837d58d1c615c28e57829460b551ff11bc296" gracePeriod=45 Nov 02 15:39:03.710582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.710497 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" containerID="cri-o://d77d098b0b63910f1b41263e709650325cb4d09be2db00cf1850b2621b915d0f" gracePeriod=45 Nov 02 15:39:03.780561 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.780528 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt"] Nov 02 15:39:03.803488 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:39:03.803449 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43587417_a6b1_4c85_8bed_0322bd778470.slice/crio-b82ab018b2e10364709e792ff55ef47732b16b103e5ac5281be2866954f8ea77 WatchSource:0}: Error finding container b82ab018b2e10364709e792ff55ef47732b16b103e5ac5281be2866954f8ea77: Status 404 returned error can't find the container with id b82ab018b2e10364709e792ff55ef47732b16b103e5ac5281be2866954f8ea77 Nov 02 15:39:03.848885 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.848804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" event={"ID":"43587417-a6b1-4c85-8bed-0322bd778470","Type":"ContainerStarted","Data":"b82ab018b2e10364709e792ff55ef47732b16b103e5ac5281be2866954f8ea77"} Nov 02 15:39:03.852797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.852694 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" event={"ID":"4c006b75-3951-4ccb-afe4-e685f98dda8e","Type":"ContainerStarted","Data":"0080de45316ef25637436e797d7ba02282a607a18410be34221d61957e32be76"} Nov 02 15:39:03.855248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.855104 5091 generic.go:356] "Generic (PLEG): container finished" podID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerID="c807b53a2c7699021117f3f29bb837d58d1c615c28e57829460b551ff11bc296" exitCode=0 Nov 02 15:39:03.855248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.855178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" event={"ID":"ea97b0cb-2f77-4da3-a23c-1a596c2d706d","Type":"ContainerDied","Data":"c807b53a2c7699021117f3f29bb837d58d1c615c28e57829460b551ff11bc296"} Nov 02 15:39:03.857242 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.857175 5091 generic.go:356] "Generic (PLEG): container finished" podID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerID="d77d098b0b63910f1b41263e709650325cb4d09be2db00cf1850b2621b915d0f" exitCode=0 Nov 02 15:39:03.857242 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.857207 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" event={"ID":"f66b777c-1cfc-4dff-add8-63e95b1ac41b","Type":"ContainerDied","Data":"d77d098b0b63910f1b41263e709650325cb4d09be2db00cf1850b2621b915d0f"} Nov 02 15:39:03.998576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:03.998494 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:39:04.080198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.080104 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:39:04.132697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.132658 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs\") pod \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " Nov 02 15:39:04.133010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.132978 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home\") pod \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " Nov 02 15:39:04.133279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.133226 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8\" (UniqueName: \"kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8\") pod \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " Nov 02 15:39:04.134471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.134408 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache\") pod \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " Nov 02 15:39:04.134809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.134785 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir\") pod \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " Nov 02 15:39:04.135154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.135130 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm\") pod \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\" (UID: \"ea97b0cb-2f77-4da3-a23c-1a596c2d706d\") " Nov 02 15:39:04.136980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.136877 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache" pod "ea97b0cb-2f77-4da3-a23c-1a596c2d706d" (UID: "ea97b0cb-2f77-4da3-a23c-1a596c2d706d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.137146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.137120 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir" pod "ea97b0cb-2f77-4da3-a23c-1a596c2d706d" (UID: "ea97b0cb-2f77-4da3-a23c-1a596c2d706d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.139216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.139055 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home" pod "ea97b0cb-2f77-4da3-a23c-1a596c2d706d" (UID: "ea97b0cb-2f77-4da3-a23c-1a596c2d706d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.139216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.139125 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8" pod "ea97b0cb-2f77-4da3-a23c-1a596c2d706d" (UID: "ea97b0cb-2f77-4da3-a23c-1a596c2d706d"). InnerVolumeSpecName "kube-api-access-cbcc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:04.139877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.139818 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm" pod "ea97b0cb-2f77-4da3-a23c-1a596c2d706d" (UID: "ea97b0cb-2f77-4da3-a23c-1a596c2d706d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.142129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.142095 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs" pod "ea97b0cb-2f77-4da3-a23c-1a596c2d706d" (UID: "ea97b0cb-2f77-4da3-a23c-1a596c2d706d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.237113 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm\" (UniqueName: \"kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm\") pod \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.237218 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home\") pod \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.237285 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache\") pod \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.237350 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm\") pod \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.237393 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir\") pod \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.237510 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs\") pod \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\" (UID: \"f66b777c-1cfc-4dff-add8-63e95b1ac41b\") " Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238177 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238199 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238215 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238227 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238240 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238256 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcc8\" (UniqueName: \"kubernetes.io/projected/ea97b0cb-2f77-4da3-a23c-1a596c2d706d-kube-api-access-cbcc8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.238337 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238222 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir" pod "f66b777c-1cfc-4dff-add8-63e95b1ac41b" (UID: "f66b777c-1cfc-4dff-add8-63e95b1ac41b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.239258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238506 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home" pod "f66b777c-1cfc-4dff-add8-63e95b1ac41b" (UID: "f66b777c-1cfc-4dff-add8-63e95b1ac41b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.239258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.238883 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache" pod "f66b777c-1cfc-4dff-add8-63e95b1ac41b" (UID: "f66b777c-1cfc-4dff-add8-63e95b1ac41b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.241801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.241774 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm" pod "f66b777c-1cfc-4dff-add8-63e95b1ac41b" (UID: "f66b777c-1cfc-4dff-add8-63e95b1ac41b"). InnerVolumeSpecName "kube-api-access-7nmcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:04.247640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.247602 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm" pod "f66b777c-1cfc-4dff-add8-63e95b1ac41b" (UID: "f66b777c-1cfc-4dff-add8-63e95b1ac41b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:04.247751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.247649 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs" pod "f66b777c-1cfc-4dff-add8-63e95b1ac41b" (UID: "f66b777c-1cfc-4dff-add8-63e95b1ac41b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:04.339280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.339247 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nmcm\" (UniqueName: \"kubernetes.io/projected/f66b777c-1cfc-4dff-add8-63e95b1ac41b-kube-api-access-7nmcm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.339280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.339277 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.339280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.339288 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.339597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.339298 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.339597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.339307 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.339597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.339315 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f66b777c-1cfc-4dff-add8-63e95b1ac41b-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:04.871654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.871600 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" event={"ID":"43587417-a6b1-4c85-8bed-0322bd778470","Type":"ContainerStarted","Data":"1b9ff189072905d0902a1245478f649e1bb513c7b6b20f46ee1312f30eedd690"} Nov 02 15:39:04.875004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.874970 5091 generic.go:356] "Generic (PLEG): container finished" podID="4c006b75-3951-4ccb-afe4-e685f98dda8e" containerID="cc0401aec53c734fc0a980e8d17e9f39784e6fcd716d9050a65ccdaad6f7126a" exitCode=0 Nov 02 15:39:04.875225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.875206 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" event={"ID":"4c006b75-3951-4ccb-afe4-e685f98dda8e","Type":"ContainerDied","Data":"cc0401aec53c734fc0a980e8d17e9f39784e6fcd716d9050a65ccdaad6f7126a"} Nov 02 15:39:04.880550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.877165 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" Nov 02 15:39:04.880550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.877229 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs" event={"ID":"ea97b0cb-2f77-4da3-a23c-1a596c2d706d","Type":"ContainerDied","Data":"9c1563dfb1fa208a2a87a53816473011fb65e1a27f08f4ea384feb82d40d5d10"} Nov 02 15:39:04.880550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.877269 5091 scope.go:122] "RemoveContainer" containerID="c807b53a2c7699021117f3f29bb837d58d1c615c28e57829460b551ff11bc296" Nov 02 15:39:04.880550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.880250 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" event={"ID":"f66b777c-1cfc-4dff-add8-63e95b1ac41b","Type":"ContainerDied","Data":"0cc111df73ed5bd38cfad8c2f9ebf083b93a420829c0b29ebce639388d065e99"} Nov 02 15:39:04.880550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.880347 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq" Nov 02 15:39:04.898903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.898871 5091 scope.go:122] "RemoveContainer" containerID="d77d098b0b63910f1b41263e709650325cb4d09be2db00cf1850b2621b915d0f" Nov 02 15:39:04.995051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:04.994984 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" podStartSLOduration=2.994965364 podStartE2EDuration="2.994965364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:04.990478582 +0000 UTC m=+2499.155031151" watchObservedRunningTime="2026-11-02 15:39:04.994965364 +0000 UTC m=+2499.159517902" Nov 02 15:39:05.210740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.210703 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq"] Nov 02 15:39:05.214982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.214953 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/simulator-w0-61a835-kserve-59dc5db855-5rswq"] Nov 02 15:39:05.320446 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.320414 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs"] Nov 02 15:39:05.324712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.324680 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-61a835/premium-w0-61a835-kserve-65d54dcf94-6tkqs"] Nov 02 15:39:05.646016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.645992 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:39:05.683054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.683031 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:39:05.765902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.765870 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.765919 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.765954 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766001 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766027 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766185 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766238 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766219 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766356 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766296 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766356 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766388 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766434 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766490 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766591 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766613 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.766648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766621 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766744 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766726 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766837 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket\") pod \"7c9cea57-bd65-4de9-b951-662b2347149a\" (UID: \"7c9cea57-bd65-4de9-b951-662b2347149a\") " Nov 02 15:39:05.766906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766873 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw\") pod \"e8f3f38d-1735-47ff-a556-05cae9593a0f\" (UID: \"e8f3f38d-1735-47ff-a556-05cae9593a0f\") " Nov 02 15:39:05.766958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.766907 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.767416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.767394 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c9cea57-bd65-4de9-b951-662b2347149a-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.767416 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.767417 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.767677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.767656 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.767779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.767754 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.767929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.767892 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.767929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.767922 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.769167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.768581 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.769167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.768594 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.769167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.768769 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.769167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.769036 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:05.769167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.769111 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.769698 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.769661 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.770221 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.770186 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:05.770221 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.770192 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:05.770556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.770493 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.771251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.771226 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.771744 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.771720 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw" pod "e8f3f38d-1735-47ff-a556-05cae9593a0f" (UID: "e8f3f38d-1735-47ff-a556-05cae9593a0f"). InnerVolumeSpecName "kube-api-access-ww7mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.772078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.772052 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c" pod "7c9cea57-bd65-4de9-b951-662b2347149a" (UID: "7c9cea57-bd65-4de9-b951-662b2347149a"). InnerVolumeSpecName "kube-api-access-p744c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:05.868660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868623 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.868660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868649 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.868660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868659 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.868660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868670 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868680 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868689 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8f3f38d-1735-47ff-a556-05cae9593a0f-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868698 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868707 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868716 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868724 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c9cea57-bd65-4de9-b951-662b2347149a-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868732 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868740 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8f3f38d-1735-47ff-a556-05cae9593a0f-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868749 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p744c\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-kube-api-access-p744c\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868757 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c9cea57-bd65-4de9-b951-662b2347149a-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868765 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c9cea57-bd65-4de9-b951-662b2347149a-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.869014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.868774 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7mw\" (UniqueName: \"kubernetes.io/projected/e8f3f38d-1735-47ff-a556-05cae9593a0f-kube-api-access-ww7mw\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:05.893270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.893229 5091 generic.go:356] "Generic (PLEG): container finished" podID="7c9cea57-bd65-4de9-b951-662b2347149a" containerID="57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b" exitCode=0 Nov 02 15:39:05.893450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.893276 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" Nov 02 15:39:05.893450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.893308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" event={"ID":"7c9cea57-bd65-4de9-b951-662b2347149a","Type":"ContainerDied","Data":"57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b"} Nov 02 15:39:05.893450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.893362 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4" event={"ID":"7c9cea57-bd65-4de9-b951-662b2347149a","Type":"ContainerDied","Data":"2c51b7ea0c628f955ccadbcb5c3c04dadaaa404f38c4aeac7830da52967204d6"} Nov 02 15:39:05.893450 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.893393 5091 scope.go:122] "RemoveContainer" containerID="57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b" Nov 02 15:39:05.895355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.895153 5091 generic.go:356] "Generic (PLEG): container finished" podID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerID="fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63" exitCode=0 Nov 02 15:39:05.895355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.895185 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" event={"ID":"e8f3f38d-1735-47ff-a556-05cae9593a0f","Type":"ContainerDied","Data":"fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63"} Nov 02 15:39:05.895355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.895217 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" Nov 02 15:39:05.895355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.895240 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj" event={"ID":"e8f3f38d-1735-47ff-a556-05cae9593a0f","Type":"ContainerDied","Data":"b4186ee4cbfdb2f8a19f92f4861938b8dba42791c62a726da7fa273cf5a6cfda"} Nov 02 15:39:05.900764 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.900741 5091 generic.go:356] "Generic (PLEG): container finished" podID="43587417-a6b1-4c85-8bed-0322bd778470" containerID="1b9ff189072905d0902a1245478f649e1bb513c7b6b20f46ee1312f30eedd690" exitCode=0 Nov 02 15:39:05.901070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.901045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" event={"ID":"43587417-a6b1-4c85-8bed-0322bd778470","Type":"ContainerDied","Data":"1b9ff189072905d0902a1245478f649e1bb513c7b6b20f46ee1312f30eedd690"} Nov 02 15:39:05.907901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.907883 5091 scope.go:122] "RemoveContainer" containerID="57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b" Nov 02 15:39:05.908284 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:05.908163 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b\": container with ID starting with 57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b not found: ID does not exist" containerID="57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b" Nov 02 15:39:05.908284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.908194 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b"} err="failed to get container status \"57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b\": rpc error: code = NotFound desc = could not find container \"57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b\": container with ID starting with 57d5115b0e1c08456fc938364d5edd4b977c3d82f7032ea4c8131c733a24182b not found: ID does not exist" Nov 02 15:39:05.908284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.908215 5091 scope.go:122] "RemoveContainer" containerID="fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63" Nov 02 15:39:05.922028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.922003 5091 scope.go:122] "RemoveContainer" containerID="fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63" Nov 02 15:39:05.923217 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:05.922400 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63\": container with ID starting with fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63 not found: ID does not exist" containerID="fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63" Nov 02 15:39:05.923217 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.922435 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63"} err="failed to get container status \"fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63\": rpc error: code = NotFound desc = could not find container \"fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63\": container with ID starting with fe648f626d2391463fb0463c23e3562cf5c7ac6135f60da3c9e1df2929e15c63 not found: ID does not exist" Nov 02 15:39:05.963765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.963685 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" path="/var/lib/kubelet/pods/ea97b0cb-2f77-4da3-a23c-1a596c2d706d/volumes" Nov 02 15:39:05.964397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:05.964373 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" path="/var/lib/kubelet/pods/f66b777c-1cfc-4dff-add8-63e95b1ac41b/volumes" Nov 02 15:39:06.113077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.113044 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:06.175003 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.174970 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj"] Nov 02 15:39:06.180248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.180218 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-7btfj"] Nov 02 15:39:06.278668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.278567 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t\" (UniqueName: \"kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t\") pod \"4c006b75-3951-4ccb-afe4-e685f98dda8e\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " Nov 02 15:39:06.279126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.278764 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca\") pod \"4c006b75-3951-4ccb-afe4-e685f98dda8e\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " Nov 02 15:39:06.279126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.278818 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca\") pod \"4c006b75-3951-4ccb-afe4-e685f98dda8e\" (UID: \"4c006b75-3951-4ccb-afe4-e685f98dda8e\") " Nov 02 15:39:06.279673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.279641 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca" pod "4c006b75-3951-4ccb-afe4-e685f98dda8e" (UID: "4c006b75-3951-4ccb-afe4-e685f98dda8e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:06.282460 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.282430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca" pod "4c006b75-3951-4ccb-afe4-e685f98dda8e" (UID: "4c006b75-3951-4ccb-afe4-e685f98dda8e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:06.284043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.284018 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t" pod "4c006b75-3951-4ccb-afe4-e685f98dda8e" (UID: "4c006b75-3951-4ccb-afe4-e685f98dda8e"). InnerVolumeSpecName "kube-api-access-ff22t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:06.284211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.284066 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4"] Nov 02 15:39:06.288097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.288074 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5aeac88e-openshift-default-789d4fdcbf-dr7j4"] Nov 02 15:39:06.380160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.380126 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff22t\" (UniqueName: \"kubernetes.io/projected/4c006b75-3951-4ccb-afe4-e685f98dda8e-kube-api-access-ff22t\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:06.380160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.380158 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:06.380160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.380168 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c006b75-3951-4ccb-afe4-e685f98dda8e-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:06.909432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.909405 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" Nov 02 15:39:06.909688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.909438 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx" event={"ID":"4c006b75-3951-4ccb-afe4-e685f98dda8e","Type":"ContainerDied","Data":"0080de45316ef25637436e797d7ba02282a607a18410be34221d61957e32be76"} Nov 02 15:39:06.909688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:06.909535 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0080de45316ef25637436e797d7ba02282a607a18410be34221d61957e32be76" Nov 02 15:39:07.063946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.063920 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:07.190616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.190000 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca\") pod \"43587417-a6b1-4c85-8bed-0322bd778470\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " Nov 02 15:39:07.190616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.190221 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca\") pod \"43587417-a6b1-4c85-8bed-0322bd778470\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " Nov 02 15:39:07.190616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.190406 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92\" (UniqueName: \"kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92\") pod \"43587417-a6b1-4c85-8bed-0322bd778470\" (UID: \"43587417-a6b1-4c85-8bed-0322bd778470\") " Nov 02 15:39:07.190616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.190406 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca" pod "43587417-a6b1-4c85-8bed-0322bd778470" (UID: "43587417-a6b1-4c85-8bed-0322bd778470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:07.191373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.191342 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:07.193995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.193967 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca" pod "43587417-a6b1-4c85-8bed-0322bd778470" (UID: "43587417-a6b1-4c85-8bed-0322bd778470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:07.195283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.195240 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92" pod "43587417-a6b1-4c85-8bed-0322bd778470" (UID: "43587417-a6b1-4c85-8bed-0322bd778470"). InnerVolumeSpecName "kube-api-access-6xf92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:07.293026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.292994 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xf92\" (UniqueName: \"kubernetes.io/projected/43587417-a6b1-4c85-8bed-0322bd778470-kube-api-access-6xf92\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:07.293026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.293025 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43587417-a6b1-4c85-8bed-0322bd778470-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:07.918836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.918799 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" Nov 02 15:39:07.918836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.918827 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt" event={"ID":"43587417-a6b1-4c85-8bed-0322bd778470","Type":"ContainerDied","Data":"b82ab018b2e10364709e792ff55ef47732b16b103e5ac5281be2866954f8ea77"} Nov 02 15:39:07.919058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.918882 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b82ab018b2e10364709e792ff55ef47732b16b103e5ac5281be2866954f8ea77" Nov 02 15:39:07.960917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.960884 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" path="/var/lib/kubelet/pods/7c9cea57-bd65-4de9-b951-662b2347149a/volumes" Nov 02 15:39:07.961647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:07.961627 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" path="/var/lib/kubelet/pods/e8f3f38d-1735-47ff-a556-05cae9593a0f/volumes" Nov 02 15:39:08.105369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:08.105307 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" containerName="payload-processing" containerID="cri-o://963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968" gracePeriod=25 Nov 02 15:39:08.207008 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:08.206921 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerName="payload-pre-processing" containerID="cri-o://a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6" gracePeriod=25 Nov 02 15:39:11.009682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.009643 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" containerID="cri-o://cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee" gracePeriod=45 Nov 02 15:39:11.290923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.290898 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:39:11.447549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447501 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir\") pod \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " Nov 02 15:39:11.447736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs\" (UniqueName: \"kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs\") pod \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " Nov 02 15:39:11.447736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447683 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache\") pod \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " Nov 02 15:39:11.447736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447732 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm\") pod \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " Nov 02 15:39:11.447895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447789 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm\" (UniqueName: \"kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm\") pod \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " Nov 02 15:39:11.447895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir" pod "17d0d0b6-8ac6-4c8a-8784-d509c9838c18" (UID: "17d0d0b6-8ac6-4c8a-8784-d509c9838c18"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.448009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.447962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home\") pod \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\" (UID: \"17d0d0b6-8ac6-4c8a-8784-d509c9838c18\") " Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.448120 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache" pod "17d0d0b6-8ac6-4c8a-8784-d509c9838c18" (UID: "17d0d0b6-8ac6-4c8a-8784-d509c9838c18"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.448376 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home" pod "17d0d0b6-8ac6-4c8a-8784-d509c9838c18" (UID: "17d0d0b6-8ac6-4c8a-8784-d509c9838c18"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.448587 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.448608 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.448622 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.450824 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs" pod "17d0d0b6-8ac6-4c8a-8784-d509c9838c18" (UID: "17d0d0b6-8ac6-4c8a-8784-d509c9838c18"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:11.451413 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.451101 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm" pod "17d0d0b6-8ac6-4c8a-8784-d509c9838c18" (UID: "17d0d0b6-8ac6-4c8a-8784-d509c9838c18"). InnerVolumeSpecName "kube-api-access-g5xrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:11.458136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.458106 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm" pod "17d0d0b6-8ac6-4c8a-8784-d509c9838c18" (UID: "17d0d0b6-8ac6-4c8a-8784-d509c9838c18"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:11.550487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.550404 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.550487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.550434 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.550487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.550444 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5xrm\" (UniqueName: \"kubernetes.io/projected/17d0d0b6-8ac6-4c8a-8784-d509c9838c18-kube-api-access-g5xrm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:11.949658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.949615 5091 generic.go:356] "Generic (PLEG): container finished" podID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerID="cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee" exitCode=0 Nov 02 15:39:11.949878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.949676 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" Nov 02 15:39:11.949878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.949711 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" event={"ID":"17d0d0b6-8ac6-4c8a-8784-d509c9838c18","Type":"ContainerDied","Data":"cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee"} Nov 02 15:39:11.949878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.949759 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq" event={"ID":"17d0d0b6-8ac6-4c8a-8784-d509c9838c18","Type":"ContainerDied","Data":"77c32135c4a1c7937e5110a942e9ef3c2b388a0a0e5390d7cb8ed73e9bbd710d"} Nov 02 15:39:11.949878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.949782 5091 scope.go:122] "RemoveContainer" containerID="cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee" Nov 02 15:39:11.967581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.967486 5091 scope.go:122] "RemoveContainer" containerID="cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee" Nov 02 15:39:11.967856 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:11.967836 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee\": container with ID starting with cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee not found: ID does not exist" containerID="cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee" Nov 02 15:39:11.967934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:11.967863 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee"} err="failed to get container status \"cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee\": rpc error: code = NotFound desc = could not find container \"cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee\": container with ID starting with cacfc8938b538b25adfd3f7f02e7c0b6f0b1bce7b9d244cf81d6e9301fafd5ee not found: ID does not exist" Nov 02 15:39:12.108942 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:12.108911 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq"] Nov 02 15:39:12.112054 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:12.112029 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-71d1eb/simulator-w3-71d1eb-kserve-5457877df4-582mq"] Nov 02 15:39:13.180158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:13.180125 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:13.180158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:13.180142 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:13.960720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:13.960688 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" path="/var/lib/kubelet/pods/17d0d0b6-8ac6-4c8a-8784-d509c9838c18/volumes" Nov 02 15:39:19.147442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:19.147407 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq"] Nov 02 15:39:19.147874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:19.147635 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" containerID="cri-o://188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79" gracePeriod=30 Nov 02 15:39:22.801469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:22.801437 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:22.801872 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:22.801531 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:23.179900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:23.179862 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:23.180091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:23.179898 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:39:24.424462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.424409 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:39:24.523684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523652 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.523976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523738 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.523976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523778 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.523976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523834 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.523976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523883 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.523976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523953 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.524239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523983 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.524239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.523978 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:24.524239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524000 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:24.524239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524045 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.524239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524172 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:24.524239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524193 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs\") pod \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\" (UID: \"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd\") " Nov 02 15:39:24.524552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524328 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:24.524552 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524411 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:24.524776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524756 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.524850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524782 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.524850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524797 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.524850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524810 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.524850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.524824 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.526254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.526201 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:24.526379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.526271 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "kube-api-access-b6kss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:24.526575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.526554 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:24.526778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.526742 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy" pod "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" (UID: "c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:24.635125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.633994 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.635125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.634033 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.635125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.634064 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kss\" (UniqueName: \"kubernetes.io/projected/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-kube-api-access-b6kss\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:24.635125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:24.634083 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:25.042443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.042402 5091 generic.go:356] "Generic (PLEG): container finished" podID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerID="188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79" exitCode=0 Nov 02 15:39:25.042676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.042451 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" Nov 02 15:39:25.042676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.042477 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" event={"ID":"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd","Type":"ContainerDied","Data":"188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79"} Nov 02 15:39:25.042676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.042553 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq" event={"ID":"c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd","Type":"ContainerDied","Data":"0f5bbd88bcd8eb37856a018cca656a8de80e00e766624db5abcfa45401186176"} Nov 02 15:39:25.042676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.042580 5091 scope.go:122] "RemoveContainer" containerID="188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79" Nov 02 15:39:25.055600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.055579 5091 scope.go:122] "RemoveContainer" containerID="188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79" Nov 02 15:39:25.055863 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:25.055842 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79\": container with ID starting with 188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79 not found: ID does not exist" containerID="188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79" Nov 02 15:39:25.055940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.055874 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79"} err="failed to get container status \"188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79\": rpc error: code = NotFound desc = could not find container \"188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79\": container with ID starting with 188c1a7bbbbab70af01848731410ec775b433006b22d06f8f9cafe63e5c57e79 not found: ID does not exist" Nov 02 15:39:25.210157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.210123 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq"] Nov 02 15:39:25.251225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.251195 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-qvwbq"] Nov 02 15:39:25.963874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:25.963835 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" path="/var/lib/kubelet/pods/c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd/volumes" Nov 02 15:39:27.982687 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:27.982641 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:39:28.429693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.429668 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:39:28.511038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.511008 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:39:28.515510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.515487 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume\") pod \"e912adb2-3cbb-4346-a85f-f05e46536ae8\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " Nov 02 15:39:28.515638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.515541 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55\" (UniqueName: \"kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55\") pod \"e912adb2-3cbb-4346-a85f-f05e46536ae8\" (UID: \"e912adb2-3cbb-4346-a85f-f05e46536ae8\") " Nov 02 15:39:28.515679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.515636 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm\" (UniqueName: \"kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm\") pod \"379c95b5-c394-4866-af0b-b5cc338d0543\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " Nov 02 15:39:28.515768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.515752 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume\") pod \"379c95b5-c394-4866-af0b-b5cc338d0543\" (UID: \"379c95b5-c394-4866-af0b-b5cc338d0543\") " Nov 02 15:39:28.515886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.515860 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume" pod "e912adb2-3cbb-4346-a85f-f05e46536ae8" (UID: "e912adb2-3cbb-4346-a85f-f05e46536ae8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:28.516133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.516113 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume" pod "379c95b5-c394-4866-af0b-b5cc338d0543" (UID: "379c95b5-c394-4866-af0b-b5cc338d0543"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:28.516286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.516245 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/379c95b5-c394-4866-af0b-b5cc338d0543-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:28.516286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.516266 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e912adb2-3cbb-4346-a85f-f05e46536ae8-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:28.517655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.517633 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55" pod "e912adb2-3cbb-4346-a85f-f05e46536ae8" (UID: "e912adb2-3cbb-4346-a85f-f05e46536ae8"). InnerVolumeSpecName "kube-api-access-tcx55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:28.517724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.517706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm" pod "379c95b5-c394-4866-af0b-b5cc338d0543" (UID: "379c95b5-c394-4866-af0b-b5cc338d0543"). InnerVolumeSpecName "kube-api-access-tvxkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:28.617147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.617111 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcx55\" (UniqueName: \"kubernetes.io/projected/e912adb2-3cbb-4346-a85f-f05e46536ae8-kube-api-access-tcx55\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:28.617147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:28.617143 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvxkm\" (UniqueName: \"kubernetes.io/projected/379c95b5-c394-4866-af0b-b5cc338d0543-kube-api-access-tvxkm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:29.103311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.103267 5091 generic.go:356] "Generic (PLEG): container finished" podID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerID="a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6" exitCode=0 Nov 02 15:39:29.103821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.103328 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" Nov 02 15:39:29.103821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.103361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" event={"ID":"e912adb2-3cbb-4346-a85f-f05e46536ae8","Type":"ContainerDied","Data":"a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6"} Nov 02 15:39:29.103821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.103437 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw" event={"ID":"e912adb2-3cbb-4346-a85f-f05e46536ae8","Type":"ContainerDied","Data":"8019e3de611a5927c4519688ec7430a225fe5870b73234d6c9cba15a3a53d20b"} Nov 02 15:39:29.103821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.103473 5091 scope.go:122] "RemoveContainer" containerID="a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6" Nov 02 15:39:29.105869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.105842 5091 generic.go:356] "Generic (PLEG): container finished" podID="379c95b5-c394-4866-af0b-b5cc338d0543" containerID="963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968" exitCode=0 Nov 02 15:39:29.106009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.105889 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" event={"ID":"379c95b5-c394-4866-af0b-b5cc338d0543","Type":"ContainerDied","Data":"963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968"} Nov 02 15:39:29.106009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.105929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" event={"ID":"379c95b5-c394-4866-af0b-b5cc338d0543","Type":"ContainerDied","Data":"52ba93d26f968b7dfc1de7ac4196485d0b254a3f67e7ee7d580b1dd24b20887b"} Nov 02 15:39:29.106009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.105893 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6" Nov 02 15:39:29.116907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.116886 5091 scope.go:122] "RemoveContainer" containerID="a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6" Nov 02 15:39:29.117279 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:29.117249 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6\": container with ID starting with a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6 not found: ID does not exist" containerID="a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6" Nov 02 15:39:29.117564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.117495 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6"} err="failed to get container status \"a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6\": rpc error: code = NotFound desc = could not find container \"a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6\": container with ID starting with a4afab3a4edbf405096aeb7ac7938c642c23e47dc2b98985d1f7923cb7b581d6 not found: ID does not exist" Nov 02 15:39:29.117564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.117539 5091 scope.go:122] "RemoveContainer" containerID="963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968" Nov 02 15:39:29.142248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.142224 5091 scope.go:122] "RemoveContainer" containerID="963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968" Nov 02 15:39:29.142608 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:29.142582 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968\": container with ID starting with 963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968 not found: ID does not exist" containerID="963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968" Nov 02 15:39:29.142714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.142612 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968"} err="failed to get container status \"963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968\": rpc error: code = NotFound desc = could not find container \"963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968\": container with ID starting with 963fcc7150e5424297a7a408174dca9dba6eaf1a6d1f7819e7b597e65c0d9968 not found: ID does not exist" Nov 02 15:39:29.275809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.275776 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6"] Nov 02 15:39:29.288541 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.281101 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-71d1eb-7c6cb85b9-tntx6"] Nov 02 15:39:29.402545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.399924 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw"] Nov 02 15:39:29.411548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.411495 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-71d1eb-7b6848c87-557pw"] Nov 02 15:39:29.628974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.628940 5091 scope.go:122] "RemoveContainer" containerID="447a3384bf7ed5248d62d0264797f0898197e5a262aea6a4af28ea3c14c55de0" Nov 02 15:39:29.648709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.648682 5091 scope.go:122] "RemoveContainer" containerID="d813821965d92cd11ef12e6f72a34045996d74c17030d11e8643fa127f57b540" Nov 02 15:39:29.999622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:29.999150 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" path="/var/lib/kubelet/pods/379c95b5-c394-4866-af0b-b5cc338d0543/volumes" Nov 02 15:39:30.000255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:30.000232 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" path="/var/lib/kubelet/pods/e912adb2-3cbb-4346-a85f-f05e46536ae8/volumes" Nov 02 15:39:34.388304 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:34.388264 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld"] Nov 02 15:39:34.389293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:34.389126 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" containerID="cri-o://9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c" gracePeriod=30 Nov 02 15:39:36.797710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:36.797675 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:39:36.798098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:36.797733 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:39:37.975125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:37.975086 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn"] Nov 02 15:39:37.975610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:37.975310 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" containerID="cri-o://a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a" gracePeriod=30 Nov 02 15:39:39.653965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.653900 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:39:39.688548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688499 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.688710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688608 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.688710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688680 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.688886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688735 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.688886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688794 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.688886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688812 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.689082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688886 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.689082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.688963 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.689082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689057 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.689250 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689137 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data\") pod \"0aa62505-cde6-4da5-821a-93400bef0cfd\" (UID: \"0aa62505-cde6-4da5-821a-93400bef0cfd\") " Nov 02 15:39:39.689307 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689267 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.689725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689697 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.689847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689813 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.689847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689836 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.689968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.689874 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.690027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.690000 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:39.691070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.691034 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:39.691180 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.691048 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:39.691655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.691633 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "kube-api-access-g75x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:39.692146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.692127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token" pod "0aa62505-cde6-4da5-821a-93400bef0cfd" (UID: "0aa62505-cde6-4da5-821a-93400bef0cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:39.791955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.791905 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.791955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.791952 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.792274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.791967 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.792274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.791980 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0aa62505-cde6-4da5-821a-93400bef0cfd-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.792274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.791994 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75x7\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-kube-api-access-g75x7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.792274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.792011 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:39.792274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:39.792023 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0aa62505-cde6-4da5-821a-93400bef0cfd-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:40.231364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.231275 5091 generic.go:356] "Generic (PLEG): container finished" podID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerID="9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c" exitCode=0 Nov 02 15:39:40.231364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.231341 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" Nov 02 15:39:40.231364 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.231350 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" event={"ID":"0aa62505-cde6-4da5-821a-93400bef0cfd","Type":"ContainerDied","Data":"9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c"} Nov 02 15:39:40.231625 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.231406 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld" event={"ID":"0aa62505-cde6-4da5-821a-93400bef0cfd","Type":"ContainerDied","Data":"04640d5cc45c2ffd5f68fcfdea458ee22a3e8b696ee75240a745253fc30a81c3"} Nov 02 15:39:40.231625 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.231429 5091 scope.go:122] "RemoveContainer" containerID="9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c" Nov 02 15:39:40.241729 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.241712 5091 scope.go:122] "RemoveContainer" containerID="9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c" Nov 02 15:39:40.242016 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:40.241992 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c\": container with ID starting with 9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c not found: ID does not exist" containerID="9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c" Nov 02 15:39:40.242111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.242023 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c"} err="failed to get container status \"9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c\": rpc error: code = NotFound desc = could not find container \"9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c\": container with ID starting with 9162cf7e405115e399a1d44f1958e599c0aa1df3f77b230dfc609e31b92c650c not found: ID does not exist" Nov 02 15:39:40.391604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.391569 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld"] Nov 02 15:39:40.396235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:40.396209 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8a2e6f-openshift-default-784945569c-nv7ld"] Nov 02 15:39:41.963971 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:41.963930 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" path="/var/lib/kubelet/pods/0aa62505-cde6-4da5-821a-93400bef0cfd/volumes" Nov 02 15:39:43.046859 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:43.046828 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca67de83_e0b6_4d04_bd24_92d6ee567b05.slice/crio-a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca67de83_e0b6_4d04_bd24_92d6ee567b05.slice/crio-conmon-a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:39:43.246171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.246111 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:39:43.255976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.255950 5091 generic.go:356] "Generic (PLEG): container finished" podID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerID="a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a" exitCode=0 Nov 02 15:39:43.256114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.256049 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" Nov 02 15:39:43.256114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.256098 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" event={"ID":"ca67de83-e0b6-4d04-bd24-92d6ee567b05","Type":"ContainerDied","Data":"a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a"} Nov 02 15:39:43.256193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.256143 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn" event={"ID":"ca67de83-e0b6-4d04-bd24-92d6ee567b05","Type":"ContainerDied","Data":"b52b3e48d83c20e98e866f4878bf4668b09c85a17fdc4765859a752dbe0f9488"} Nov 02 15:39:43.256193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.256163 5091 scope.go:122] "RemoveContainer" containerID="a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a" Nov 02 15:39:43.266021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.266005 5091 scope.go:122] "RemoveContainer" containerID="a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a" Nov 02 15:39:43.266294 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:43.266275 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a\": container with ID starting with a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a not found: ID does not exist" containerID="a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a" Nov 02 15:39:43.266369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.266303 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a"} err="failed to get container status \"a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a\": rpc error: code = NotFound desc = could not find container \"a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a\": container with ID starting with a8ed1556ab42937cddafc78a9238da632c478836ffb0dc5a030a6d474b548e1a not found: ID does not exist" Nov 02 15:39:43.334048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334019 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334091 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334178 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334264 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334302 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:43.334358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334501 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334512 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.334571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334532 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:43.335015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334972 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:43.335015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.334994 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:43.335015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335001 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b\") pod \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\" (UID: \"ca67de83-e0b6-4d04-bd24-92d6ee567b05\") " Nov 02 15:39:43.335239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335059 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:43.335481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335462 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.335566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335487 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.335566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335502 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.335566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335533 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.335566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.335547 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.336728 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.336706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:39:43.336983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.336959 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:43.337316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.337298 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "kube-api-access-7zd5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:43.337461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.337443 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo" pod "ca67de83-e0b6-4d04-bd24-92d6ee567b05" (UID: "ca67de83-e0b6-4d04-bd24-92d6ee567b05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:39:43.437389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.437354 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.437389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.437389 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.437389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.437404 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zd5b\" (UniqueName: \"kubernetes.io/projected/ca67de83-e0b6-4d04-bd24-92d6ee567b05-kube-api-access-7zd5b\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.437746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.437422 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca67de83-e0b6-4d04-bd24-92d6ee567b05-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:43.694286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.694245 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn"] Nov 02 15:39:43.703629 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.703592 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-624908-openshift-default-97c84d5d9-pfpqn"] Nov 02 15:39:43.961315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:43.961278 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" path="/var/lib/kubelet/pods/ca67de83-e0b6-4d04-bd24-92d6ee567b05/volumes" Nov 02 15:39:49.435775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.435737 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg"] Nov 02 15:39:49.436592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436574 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" Nov 02 15:39:49.436644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436595 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" Nov 02 15:39:49.436644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436622 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerName="payload-pre-processing" Nov 02 15:39:49.436644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436630 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerName="payload-pre-processing" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436652 5091 cpu_manager.go:414] "containerMap: removing container" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" containerName="payload-processing" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436660 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" containerName="payload-processing" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436669 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436677 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436689 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436696 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436709 5091 cpu_manager.go:414] "containerMap: removing container" podUID="43587417-a6b1-4c85-8bed-0322bd778470" containerName="revoke-keys" Nov 02 15:39:49.436740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436735 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43587417-a6b1-4c85-8bed-0322bd778470" containerName="revoke-keys" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436753 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436761 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436777 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436784 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436796 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4c006b75-3951-4ccb-afe4-e685f98dda8e" containerName="revoke-keys" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436803 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c006b75-3951-4ccb-afe4-e685f98dda8e" containerName="revoke-keys" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436818 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436825 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436846 5091 cpu_manager.go:414] "containerMap: removing container" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436853 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436865 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" Nov 02 15:39:49.436972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436872 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.436998 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="379c95b5-c394-4866-af0b-b5cc338d0543" containerName="payload-processing" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437010 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa62505-cde6-4da5-821a-93400bef0cfd" containerName="istio-proxy" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437021 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b25ebf-afb3-49d6-b7cb-e9eca9880ccd" containerName="istio-proxy" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437030 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="43587417-a6b1-4c85-8bed-0322bd778470" containerName="revoke-keys" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437040 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea97b0cb-2f77-4da3-a23c-1a596c2d706d" containerName="main" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437050 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca67de83-e0b6-4d04-bd24-92d6ee567b05" containerName="istio-proxy" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437060 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c006b75-3951-4ccb-afe4-e685f98dda8e" containerName="revoke-keys" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437070 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8f3f38d-1735-47ff-a556-05cae9593a0f" containerName="istio-proxy" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437080 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d0d0b6-8ac6-4c8a-8784-d509c9838c18" containerName="main" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437087 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9cea57-bd65-4de9-b951-662b2347149a" containerName="istio-proxy" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437101 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66b777c-1cfc-4dff-add8-63e95b1ac41b" containerName="main" Nov 02 15:39:49.437326 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.437110 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e912adb2-3cbb-4346-a85f-f05e46536ae8" containerName="payload-pre-processing" Nov 02 15:39:49.442585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.442561 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.474925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.474909 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg"] Nov 02 15:39:49.515563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515502 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515615 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515702 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515768 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515874 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxqz\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515903 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.515932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515923 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.516000 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.515940 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.616977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617028 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617085 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617102 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxqz\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617124 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617154 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617296 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617574 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617649 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617762 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.617836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617781 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.618002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.617978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.619532 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.619493 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.619723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.619705 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.675817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.675785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.676682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.676660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxqz\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz\") pod \"e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:49.779430 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:49.779343 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:50.218424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:50.218392 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg"] Nov 02 15:39:50.258911 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:39:50.258878 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35054c36_48a1_4344_8008_6876de3cbde1.slice/crio-490596bf4fad8b32d8119040ca4900b110fa615af411bb370bf0e64a2fc6dc41 WatchSource:0}: Error finding container 490596bf4fad8b32d8119040ca4900b110fa615af411bb370bf0e64a2fc6dc41: Status 404 returned error can't find the container with id 490596bf4fad8b32d8119040ca4900b110fa615af411bb370bf0e64a2fc6dc41 Nov 02 15:39:50.261986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:50.261944 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:50.262061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:50.262040 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:50.262106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:50.262080 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:50.311873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:50.311842 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" event={"ID":"35054c36-48a1-4344-8008-6876de3cbde1","Type":"ContainerStarted","Data":"490596bf4fad8b32d8119040ca4900b110fa615af411bb370bf0e64a2fc6dc41"} Nov 02 15:39:51.320166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.320125 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" event={"ID":"35054c36-48a1-4344-8008-6876de3cbde1","Type":"ContainerStarted","Data":"a9105b2d0488e0afd4c3da98bb1e2d1b2f5e7c39247cfe6c0f3eb816d6a50a80"} Nov 02 15:39:51.483003 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.482952 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podStartSLOduration=2.482938525 podStartE2EDuration="2.482938525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:51.48249202 +0000 UTC m=+2545.647044557" watchObservedRunningTime="2026-11-02 15:39:51.482938525 +0000 UTC m=+2545.647491065" Nov 02 15:39:51.780154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.780113 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:51.781691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.781663 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 15:39:51.781845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.781716 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 15:39:51.967286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.967251 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg"] Nov 02 15:39:51.970973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:51.970947 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.045822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.045735 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:39:52.046199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.046176 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:39:52.063617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.063582 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg"] Nov 02 15:39:52.065068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.065044 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mbxm\" (UniqueName: \"kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.065421 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.065402 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.066536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.066481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.091873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.091844 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f"] Nov 02 15:39:52.095934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.095913 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.170170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.168445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.170170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.168565 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mbxm\" (UniqueName: \"kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.170170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.168707 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.170493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.170460 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.171616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.171589 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f"] Nov 02 15:39:52.172398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.172374 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.250538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.249191 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mbxm\" (UniqueName: \"kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm\") pod \"maas-api-revoke-sub-1a6bc4f6-j22sg\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.271007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.270972 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbr2\" (UniqueName: \"kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.271187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.271073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.271187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.271104 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.285689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.285661 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:52.376540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.372720 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbr2\" (UniqueName: \"kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.376540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.372792 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.376540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.372819 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.376540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.373827 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.377182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.376828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.481966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.481930 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbr2\" (UniqueName: \"kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2\") pod \"maas-api-revoke-sub-de2f5f75-btw2f\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.710017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.709979 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:52.763205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.761932 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg"] Nov 02 15:39:52.780938 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.780834 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 15:39:52.780938 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:52.780888 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 15:39:53.141983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.131459 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f"] Nov 02 15:39:53.369593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.367467 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" event={"ID":"b301ba99-9256-44b7-8ccb-61aa17e0549d","Type":"ContainerStarted","Data":"bd260f3b8feb227f8cb66493881e2bdbbe9ce26edbc2409db2a7141b82a29e8c"} Nov 02 15:39:53.370996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.370963 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" event={"ID":"8be724fa-40b0-4d27-8287-48106f479d25","Type":"ContainerStarted","Data":"2fd61e185d3d2aca9c51619e47852ac2362f546e7d7f8e81f65072789bc6d270"} Nov 02 15:39:53.371134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.371019 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" event={"ID":"8be724fa-40b0-4d27-8287-48106f479d25","Type":"ContainerStarted","Data":"00ed7f603894bcc91bf2099bfc87ffc7852746a0fd171dccaf654813df6085bf"} Nov 02 15:39:53.486196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.486143 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" podStartSLOduration=2.486128791 podStartE2EDuration="2.486128791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:53.484055853 +0000 UTC m=+2547.648608396" watchObservedRunningTime="2026-11-02 15:39:53.486128791 +0000 UTC m=+2547.650681330" Nov 02 15:39:53.780730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.780642 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 15:39:53.780730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.780715 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 15:39:53.926169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.926136 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm"] Nov 02 15:39:53.937815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.937788 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:53.993537 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:53.988896 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e3b351-openshift-default-dockercfg-rn667\"" Nov 02 15:39:54.031587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.031099 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm"] Nov 02 15:39:54.105931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.105825 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.105931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.105884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6c5\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.105931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.105933 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.106255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.105957 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.106255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.106077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.106255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.106151 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.106255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.106199 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.107662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.106343 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.107662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.106397 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207212 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207266 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6c5\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207300 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207666 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207723 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207799 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207785 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.207960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207935 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.208111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.207983 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.208209 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.208188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.208340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.208287 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.208686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.208662 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.208766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.208662 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.210775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.210753 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.211019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.210999 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.277538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.273866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6c5\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.277538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.275221 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token\") pod \"e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.288901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.288823 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:54.383197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.382357 5091 generic.go:356] "Generic (PLEG): container finished" podID="b301ba99-9256-44b7-8ccb-61aa17e0549d" containerID="b5b2be9ef9372a1fc1729341abdde14326fe350aa6bceac6f9d61922eb5e307e" exitCode=0 Nov 02 15:39:54.383197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.382469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" event={"ID":"b301ba99-9256-44b7-8ccb-61aa17e0549d","Type":"ContainerDied","Data":"b5b2be9ef9372a1fc1729341abdde14326fe350aa6bceac6f9d61922eb5e307e"} Nov 02 15:39:54.385240 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.385211 5091 generic.go:356] "Generic (PLEG): container finished" podID="8be724fa-40b0-4d27-8287-48106f479d25" containerID="2fd61e185d3d2aca9c51619e47852ac2362f546e7d7f8e81f65072789bc6d270" exitCode=0 Nov 02 15:39:54.385358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.385261 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" event={"ID":"8be724fa-40b0-4d27-8287-48106f479d25","Type":"ContainerDied","Data":"2fd61e185d3d2aca9c51619e47852ac2362f546e7d7f8e81f65072789bc6d270"} Nov 02 15:39:54.779557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.778657 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm"] Nov 02 15:39:54.780797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.780772 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 15:39:54.780906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.780821 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 15:39:54.782641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.782091 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:54.782641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.782170 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:54.782641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:54.782211 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:39:55.395112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.395076 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" event={"ID":"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1","Type":"ContainerStarted","Data":"55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62"} Nov 02 15:39:55.395441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.395397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" event={"ID":"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1","Type":"ContainerStarted","Data":"25ba42b9a2fbe7ab55559178c208cab6b7f96bab9122d814d62c8b048780f151"} Nov 02 15:39:55.516800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.516748 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podStartSLOduration=2.516732003 podStartE2EDuration="2.516732003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:39:55.513127068 +0000 UTC m=+2549.677679609" watchObservedRunningTime="2026-11-02 15:39:55.516732003 +0000 UTC m=+2549.681284543" Nov 02 15:39:55.609554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.606393 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:55.667810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.667741 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.733011 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2\" (UniqueName: \"kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2\") pod \"b301ba99-9256-44b7-8ccb-61aa17e0549d\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.733093 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca\") pod \"8be724fa-40b0-4d27-8287-48106f479d25\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.733140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm\" (UniqueName: \"kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm\") pod \"8be724fa-40b0-4d27-8287-48106f479d25\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.733213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca\") pod \"b301ba99-9256-44b7-8ccb-61aa17e0549d\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.733341 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca\") pod \"8be724fa-40b0-4d27-8287-48106f479d25\" (UID: \"8be724fa-40b0-4d27-8287-48106f479d25\") " Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.733383 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca\") pod \"b301ba99-9256-44b7-8ccb-61aa17e0549d\" (UID: \"b301ba99-9256-44b7-8ccb-61aa17e0549d\") " Nov 02 15:39:55.738550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.737714 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca" pod "b301ba99-9256-44b7-8ccb-61aa17e0549d" (UID: "b301ba99-9256-44b7-8ccb-61aa17e0549d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:55.774877 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.774671 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2" pod "b301ba99-9256-44b7-8ccb-61aa17e0549d" (UID: "b301ba99-9256-44b7-8ccb-61aa17e0549d"). InnerVolumeSpecName "kube-api-access-zsbr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:55.775739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.775706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca" pod "8be724fa-40b0-4d27-8287-48106f479d25" (UID: "8be724fa-40b0-4d27-8287-48106f479d25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:39:55.777989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.777558 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca" pod "8be724fa-40b0-4d27-8287-48106f479d25" (UID: "8be724fa-40b0-4d27-8287-48106f479d25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:55.780685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.780120 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 15:39:55.780685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.780174 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 15:39:55.784671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.784629 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca" pod "b301ba99-9256-44b7-8ccb-61aa17e0549d" (UID: "b301ba99-9256-44b7-8ccb-61aa17e0549d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:39:55.784835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.784809 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm" pod "8be724fa-40b0-4d27-8287-48106f479d25" (UID: "8be724fa-40b0-4d27-8287-48106f479d25"). InnerVolumeSpecName "kube-api-access-2mbxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:39:55.869179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.865736 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbr2\" (UniqueName: \"kubernetes.io/projected/b301ba99-9256-44b7-8ccb-61aa17e0549d-kube-api-access-zsbr2\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:55.869179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.865773 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:55.869179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.865788 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mbxm\" (UniqueName: \"kubernetes.io/projected/8be724fa-40b0-4d27-8287-48106f479d25-kube-api-access-2mbxm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:55.869179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.865803 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:55.869179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.865817 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8be724fa-40b0-4d27-8287-48106f479d25-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:55.869179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:55.865831 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b301ba99-9256-44b7-8ccb-61aa17e0549d-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:39:56.289584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.289486 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:56.295449 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.295422 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:56.403619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.403509 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" Nov 02 15:39:56.403619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.403547 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg" event={"ID":"8be724fa-40b0-4d27-8287-48106f479d25","Type":"ContainerDied","Data":"00ed7f603894bcc91bf2099bfc87ffc7852746a0fd171dccaf654813df6085bf"} Nov 02 15:39:56.403619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.403597 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ed7f603894bcc91bf2099bfc87ffc7852746a0fd171dccaf654813df6085bf" Nov 02 15:39:56.405044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.405019 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" event={"ID":"b301ba99-9256-44b7-8ccb-61aa17e0549d","Type":"ContainerDied","Data":"bd260f3b8feb227f8cb66493881e2bdbbe9ce26edbc2409db2a7141b82a29e8c"} Nov 02 15:39:56.405167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.405064 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd260f3b8feb227f8cb66493881e2bdbbe9ce26edbc2409db2a7141b82a29e8c" Nov 02 15:39:56.405167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.405064 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f" Nov 02 15:39:56.405295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.405243 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:56.406349 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.406331 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:39:56.779776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.779738 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" start-of-body= Nov 02 15:39:56.779980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:56.779806 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.168:15021/healthz/ready\": dial tcp 10.132.0.168:15021: connect: connection refused" Nov 02 15:39:57.938198 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:39:57.938153 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:39:58.076700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:58.076665 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:58.078184 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:58.078157 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:39:58.079127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:39:58.078944 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:40:02.576886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.576852 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8"] Nov 02 15:40:02.577412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.577398 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8be724fa-40b0-4d27-8287-48106f479d25" containerName="revoke-keys" Nov 02 15:40:02.577456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.577412 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be724fa-40b0-4d27-8287-48106f479d25" containerName="revoke-keys" Nov 02 15:40:02.577456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.577448 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b301ba99-9256-44b7-8ccb-61aa17e0549d" containerName="revoke-keys" Nov 02 15:40:02.577456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.577454 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b301ba99-9256-44b7-8ccb-61aa17e0549d" containerName="revoke-keys" Nov 02 15:40:02.577596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.577551 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be724fa-40b0-4d27-8287-48106f479d25" containerName="revoke-keys" Nov 02 15:40:02.577596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.577566 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b301ba99-9256-44b7-8ccb-61aa17e0549d" containerName="revoke-keys" Nov 02 15:40:02.582254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.582232 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.636999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.636963 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8"] Nov 02 15:40:02.661510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.661465 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xg4\" (UniqueName: \"kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4\") pod \"payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.661706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.661627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.679835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.679807 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e3b351\"" Nov 02 15:40:02.680027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.679878 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e3b351-dockercfg-tf45c\"" Nov 02 15:40:02.763104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.763068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.763341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.763321 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xg4\" (UniqueName: \"kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4\") pod \"payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.763906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.763884 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.822199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.822158 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xg4\" (UniqueName: \"kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4\") pod \"payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:02.896424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:02.896393 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:03.278788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:03.278711 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8"] Nov 02 15:40:03.283465 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:40:03.283437 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cffcab8_f4c0_4d7e_ab2f_06119f06149e.slice/crio-3d7425a8e5a091956e03b8809d14c3fe372f70fa14b6e383b9f34500424bc434 WatchSource:0}: Error finding container 3d7425a8e5a091956e03b8809d14c3fe372f70fa14b6e383b9f34500424bc434: Status 404 returned error can't find the container with id 3d7425a8e5a091956e03b8809d14c3fe372f70fa14b6e383b9f34500424bc434 Nov 02 15:40:03.455623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:03.455583 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" event={"ID":"5cffcab8-f4c0-4d7e-ab2f-06119f06149e","Type":"ContainerStarted","Data":"b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685"} Nov 02 15:40:03.455796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:03.455643 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" event={"ID":"5cffcab8-f4c0-4d7e-ab2f-06119f06149e","Type":"ContainerStarted","Data":"3d7425a8e5a091956e03b8809d14c3fe372f70fa14b6e383b9f34500424bc434"} Nov 02 15:40:03.455796 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:03.455693 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:03.597281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:03.597166 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" podStartSLOduration=1.597153238 podStartE2EDuration="1.597153238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:03.59485375 +0000 UTC m=+2557.759406291" watchObservedRunningTime="2026-11-02 15:40:03.597153238 +0000 UTC m=+2557.761705830" Nov 02 15:40:14.466550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:14.466422 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:40:17.692431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:17.692392 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc"] Nov 02 15:40:17.738636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:17.738605 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-71d1eb-66lsc"] Nov 02 15:40:17.961761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:17.961664 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52a4144-a48a-4e12-88c6-978fc38c14cc" path="/var/lib/kubelet/pods/b52a4144-a48a-4e12-88c6-978fc38c14cc/volumes" Nov 02 15:40:25.407125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:25.407076 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq"] Nov 02 15:40:25.407577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:25.407322 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" containerID="cri-o://6fe38af1bf49e510cff9ce2bdf4545c98894a605fdaf2d44364e729a55ff8a2e" gracePeriod=30 Nov 02 15:40:25.420266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:25.420241 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg"] Nov 02 15:40:25.420427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:25.420408 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" containerID="cri-o://a9105b2d0488e0afd4c3da98bb1e2d1b2f5e7c39247cfe6c0f3eb816d6a50a80" gracePeriod=30 Nov 02 15:40:27.959074 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:40:27.959038 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:40:28.080549 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:28.080498 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:40:28.080713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:28.080566 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:40:29.801812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:29.801761 5091 scope.go:122] "RemoveContainer" containerID="379d0c4564617a84ee01078598b248065cc87cf36c89d4f1319c81958d4836e9" Nov 02 15:40:30.746551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.742009 5091 generic.go:356] "Generic (PLEG): container finished" podID="35054c36-48a1-4344-8008-6876de3cbde1" containerID="a9105b2d0488e0afd4c3da98bb1e2d1b2f5e7c39247cfe6c0f3eb816d6a50a80" exitCode=0 Nov 02 15:40:30.746551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.742185 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:40:30.746551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.742197 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" event={"ID":"35054c36-48a1-4344-8008-6876de3cbde1","Type":"ContainerDied","Data":"a9105b2d0488e0afd4c3da98bb1e2d1b2f5e7c39247cfe6c0f3eb816d6a50a80"} Nov 02 15:40:30.746551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.744056 5091 generic.go:356] "Generic (PLEG): container finished" podID="ab86c317-ec5b-4378-b124-3bc39162157d" containerID="6fe38af1bf49e510cff9ce2bdf4545c98894a605fdaf2d44364e729a55ff8a2e" exitCode=0 Nov 02 15:40:30.746551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.744128 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" event={"ID":"ab86c317-ec5b-4378-b124-3bc39162157d","Type":"ContainerDied","Data":"6fe38af1bf49e510cff9ce2bdf4545c98894a605fdaf2d44364e729a55ff8a2e"} Nov 02 15:40:30.746551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.744174 5091 scope.go:122] "RemoveContainer" containerID="6fe38af1bf49e510cff9ce2bdf4545c98894a605fdaf2d44364e729a55ff8a2e" Nov 02 15:40:30.792720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.792685 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:40:30.921212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921169 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921248 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921284 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921348 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921416 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921474 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921531 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921566 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921672 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.921734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921706 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921740 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921802 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921834 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.921980 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.922022 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs\") pod \"35054c36-48a1-4344-8008-6876de3cbde1\" (UID: \"35054c36-48a1-4344-8008-6876de3cbde1\") " Nov 02 15:40:30.922310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.922052 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo\") pod \"ab86c317-ec5b-4378-b124-3bc39162157d\" (UID: \"ab86c317-ec5b-4378-b124-3bc39162157d\") " Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.923395 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.923709 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.923910 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.924311 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.924491 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.925149 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.925430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.925666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.926127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.925877 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.930715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.930685 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.935797 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.935801 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.935852 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "kube-api-access-p2qw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.935902 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.935960 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.936052 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "kube-api-access-fcxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.936176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.936146 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token" pod "35054c36-48a1-4344-8008-6876de3cbde1" (UID: "35054c36-48a1-4344-8008-6876de3cbde1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:30.937081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:30.937052 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy" pod "ab86c317-ec5b-4378-b124-3bc39162157d" (UID: "ab86c317-ec5b-4378-b124-3bc39162157d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:40:31.024154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024080 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2qw8\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-kube-api-access-p2qw8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024110 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024120 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024129 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024138 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab86c317-ec5b-4378-b124-3bc39162157d-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024146 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35054c36-48a1-4344-8008-6876de3cbde1-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024159 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024176 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024188 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024197 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024206 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024214 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024222 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab86c317-ec5b-4378-b124-3bc39162157d-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024229 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxqz\" (UniqueName: \"kubernetes.io/projected/35054c36-48a1-4344-8008-6876de3cbde1-kube-api-access-fcxqz\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024237 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab86c317-ec5b-4378-b124-3bc39162157d-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024246 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35054c36-48a1-4344-8008-6876de3cbde1-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024254 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35054c36-48a1-4344-8008-6876de3cbde1-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.024448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.024262 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab86c317-ec5b-4378-b124-3bc39162157d-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:31.753970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.753886 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" Nov 02 15:40:31.753970 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.753911 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg" event={"ID":"35054c36-48a1-4344-8008-6876de3cbde1","Type":"ContainerDied","Data":"490596bf4fad8b32d8119040ca4900b110fa615af411bb370bf0e64a2fc6dc41"} Nov 02 15:40:31.754191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.754001 5091 scope.go:122] "RemoveContainer" containerID="a9105b2d0488e0afd4c3da98bb1e2d1b2f5e7c39247cfe6c0f3eb816d6a50a80" Nov 02 15:40:31.755439 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.755035 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" Nov 02 15:40:31.755439 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.755112 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq" event={"ID":"ab86c317-ec5b-4378-b124-3bc39162157d","Type":"ContainerDied","Data":"1fcae796017b91b542398e58c015c32b6fc4730fbed15c49674a655c9daa4241"} Nov 02 15:40:31.961149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:31.961117 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg"] Nov 02 15:40:32.007149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.007068 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-gvvqg"] Nov 02 15:40:32.115781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.115743 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq"] Nov 02 15:40:32.120988 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.120956 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-71d1eb-openshift-default-d4744b75f-spzvq"] Nov 02 15:40:32.764161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.764121 5091 generic.go:356] "Generic (PLEG): container finished" podID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerID="3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f" exitCode=0 Nov 02 15:40:32.764377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.764192 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" event={"ID":"36dc0efb-2f20-413e-9d8b-b26e02769233","Type":"ContainerDied","Data":"3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f"} Nov 02 15:40:32.764617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.764600 5091 scope.go:122] "RemoveContainer" containerID="3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f" Nov 02 15:40:32.766161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.766134 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:32.766233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.766203 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:32.766269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:32.766234 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:33.773165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:33.773122 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" event={"ID":"36dc0efb-2f20-413e-9d8b-b26e02769233","Type":"ContainerStarted","Data":"fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a"} Nov 02 15:40:33.963477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:33.963440 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35054c36-48a1-4344-8008-6876de3cbde1" path="/var/lib/kubelet/pods/35054c36-48a1-4344-8008-6876de3cbde1/volumes" Nov 02 15:40:33.963983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:33.963967 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" path="/var/lib/kubelet/pods/ab86c317-ec5b-4378-b124-3bc39162157d/volumes" Nov 02 15:40:34.700425 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:34.700388 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:40:34.700622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:34.700558 5091 patch_prober.go:29] interesting pod/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 15:40:34.700622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:34.700609 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 15:40:35.700604 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:35.700573 5091 patch_prober.go:29] interesting pod/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 15:40:35.700969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:35.700633 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 15:40:36.700288 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:36.700251 5091 patch_prober.go:29] interesting pod/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 15:40:36.700488 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:36.700308 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 15:40:37.670904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.670868 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt"] Nov 02 15:40:37.671530 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.671500 5091 cpu_manager.go:414] "containerMap: removing container" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" Nov 02 15:40:37.671530 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.671529 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" Nov 02 15:40:37.671690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.671601 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" Nov 02 15:40:37.671690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.671607 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" Nov 02 15:40:37.671690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.671682 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab86c317-ec5b-4378-b124-3bc39162157d" containerName="istio-proxy" Nov 02 15:40:37.671847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.671694 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="35054c36-48a1-4344-8008-6876de3cbde1" containerName="istio-proxy" Nov 02 15:40:37.676507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.676485 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.701063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.701039 5091 patch_prober.go:29] interesting pod/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 15:40:37.701183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.701083 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 15:40:37.722255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.722231 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.722352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.722275 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8p6\" (UniqueName: \"kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6\") pod \"payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.787105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.746300 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt"] Nov 02 15:40:37.787105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.772883 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fe8ef1-dockercfg-2qsfm\"" Nov 02 15:40:37.823538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.823205 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fe8ef1\"" Nov 02 15:40:37.843474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.825381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.843474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.825448 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8p6\" (UniqueName: \"kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6\") pod \"payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.857704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.856179 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.884356 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.884318 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8p6\" (UniqueName: \"kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6\") pod \"payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:37.992135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:37.992052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:38.388956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.388927 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt"] Nov 02 15:40:38.700470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.700381 5091 patch_prober.go:29] interesting pod/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 02 15:40:38.700936 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.700497 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 02 15:40:38.887697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.887642 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" event={"ID":"5fe06013-f933-4cef-8fb9-0d27846c1a01","Type":"ContainerStarted","Data":"34b7a90942ae5c7459732ff042c9e4d0ed64cfe366581df97317a0d1a0278f47"} Nov 02 15:40:38.887876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.887717 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" event={"ID":"5fe06013-f933-4cef-8fb9-0d27846c1a01","Type":"ContainerStarted","Data":"df0a9103c5d6553513dadb2cbf9052423a93b2fee4f144615703eaac40aab3f3"} Nov 02 15:40:38.887876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.887747 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:38.999941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:38.999832 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" podStartSLOduration=1.999817516 podStartE2EDuration="1.999817516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:38.997639405 +0000 UTC m=+2593.162191948" watchObservedRunningTime="2026-11-02 15:40:38.999817516 +0000 UTC m=+2593.164370049" Nov 02 15:40:39.705008 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:39.704975 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:40:39.705455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:39.705442 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:40:39.706169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:39.706145 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:40:40.285555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.285493 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx"] Nov 02 15:40:40.289742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.289716 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.341991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.341963 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:40:40.342195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.342007 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:40:40.347112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.347088 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx"] Nov 02 15:40:40.384981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.384948 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.385180 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.384999 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.385257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.385215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kppr\" (UniqueName: \"kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.486848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.486810 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kppr\" (UniqueName: \"kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.487032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.486883 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.487032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.486926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.487735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.487709 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.489309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.489275 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.545652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.545581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kppr\" (UniqueName: \"kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr\") pod \"maas-api-revoke-sub-ac60516b-fz6qx\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.603471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.603447 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:40.981468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:40.981431 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx"] Nov 02 15:40:40.986437 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:40:40.986402 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa37c4c_62ef_481a_ac39_0d6b897e5eab.slice/crio-534dc7f06f1d6aac153ec8ab946d3af8232fab98c08b2163bd251855875975d1 WatchSource:0}: Error finding container 534dc7f06f1d6aac153ec8ab946d3af8232fab98c08b2163bd251855875975d1: Status 404 returned error can't find the container with id 534dc7f06f1d6aac153ec8ab946d3af8232fab98c08b2163bd251855875975d1 Nov 02 15:40:41.909500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:41.909464 5091 generic.go:356] "Generic (PLEG): container finished" podID="6aa37c4c-62ef-481a-ac39-0d6b897e5eab" containerID="a5547735517a02d46fb47616a2acafd6200f9b7892967991dd7c75f530538b0b" exitCode=0 Nov 02 15:40:41.909669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:41.909510 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" event={"ID":"6aa37c4c-62ef-481a-ac39-0d6b897e5eab","Type":"ContainerDied","Data":"a5547735517a02d46fb47616a2acafd6200f9b7892967991dd7c75f530538b0b"} Nov 02 15:40:41.909669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:41.909584 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" event={"ID":"6aa37c4c-62ef-481a-ac39-0d6b897e5eab","Type":"ContainerStarted","Data":"534dc7f06f1d6aac153ec8ab946d3af8232fab98c08b2163bd251855875975d1"} Nov 02 15:40:43.077833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.077803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:43.120378 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.120349 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca\") pod \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " Nov 02 15:40:43.120495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.120461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr\" (UniqueName: \"kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr\") pod \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " Nov 02 15:40:43.120580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.120568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca\") pod \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\" (UID: \"6aa37c4c-62ef-481a-ac39-0d6b897e5eab\") " Nov 02 15:40:43.120797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.120769 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca" pod "6aa37c4c-62ef-481a-ac39-0d6b897e5eab" (UID: "6aa37c4c-62ef-481a-ac39-0d6b897e5eab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:43.120912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.120897 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:43.122593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.122569 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca" pod "6aa37c4c-62ef-481a-ac39-0d6b897e5eab" (UID: "6aa37c4c-62ef-481a-ac39-0d6b897e5eab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:40:43.122593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.122584 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr" pod "6aa37c4c-62ef-481a-ac39-0d6b897e5eab" (UID: "6aa37c4c-62ef-481a-ac39-0d6b897e5eab"). InnerVolumeSpecName "kube-api-access-7kppr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:43.222671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.222593 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:43.222671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.222620 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kppr\" (UniqueName: \"kubernetes.io/projected/6aa37c4c-62ef-481a-ac39-0d6b897e5eab-kube-api-access-7kppr\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:43.923574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.923538 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" Nov 02 15:40:43.923574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.923562 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx" event={"ID":"6aa37c4c-62ef-481a-ac39-0d6b897e5eab","Type":"ContainerDied","Data":"534dc7f06f1d6aac153ec8ab946d3af8232fab98c08b2163bd251855875975d1"} Nov 02 15:40:43.923844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:43.923627 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="534dc7f06f1d6aac153ec8ab946d3af8232fab98c08b2163bd251855875975d1" Nov 02 15:40:48.623820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.623775 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b"] Nov 02 15:40:48.624595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.624573 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6aa37c4c-62ef-481a-ac39-0d6b897e5eab" containerName="revoke-keys" Nov 02 15:40:48.624595 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.624594 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa37c4c-62ef-481a-ac39-0d6b897e5eab" containerName="revoke-keys" Nov 02 15:40:48.624782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.624762 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa37c4c-62ef-481a-ac39-0d6b897e5eab" containerName="revoke-keys" Nov 02 15:40:48.628469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.628448 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.684053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.684021 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b1ee37-openshift-default-dockercfg-4kghq\"" Nov 02 15:40:48.684759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.684734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.684921 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.684794 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.684921 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.684819 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.685065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.684959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.685065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.685022 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.685201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.685181 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.685318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.685303 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd854\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.685403 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.685378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.685462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.685447 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.736396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.736363 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b"] Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791245 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd854\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791372 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791409 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791462 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791492 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791547 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791593 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.791690 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.792916 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.792184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.794055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.794029 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.797049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.795566 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.797269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.796747 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.797428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.796971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.806494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.806430 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.806628 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.806603 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.852837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.852811 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd854\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.855406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.855376 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token\") pod \"e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:48.945079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:48.944987 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:49.382612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.382580 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b"] Nov 02 15:40:49.387324 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:40:49.387295 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3763b422_89cf_4336_8097_cec4de9907bd.slice/crio-c52cdf50c3be4275d4e2f224b26be57e31aa0e5acf405923ac67155221b7ab34 WatchSource:0}: Error finding container c52cdf50c3be4275d4e2f224b26be57e31aa0e5acf405923ac67155221b7ab34: Status 404 returned error can't find the container with id c52cdf50c3be4275d4e2f224b26be57e31aa0e5acf405923ac67155221b7ab34 Nov 02 15:40:49.390227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.390194 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:49.390319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.390277 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:49.390395 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.390349 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:40:49.897890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.897864 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:40:49.965865 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.965830 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" event={"ID":"3763b422-89cf-4336-8097-cec4de9907bd","Type":"ContainerStarted","Data":"95e0798e801764ce46fcfc6205278c2b03fe90b877d062d7a914dc1faaf01314"} Nov 02 15:40:49.966045 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:49.965879 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" event={"ID":"3763b422-89cf-4336-8097-cec4de9907bd","Type":"ContainerStarted","Data":"c52cdf50c3be4275d4e2f224b26be57e31aa0e5acf405923ac67155221b7ab34"} Nov 02 15:40:50.126617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:50.126565 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podStartSLOduration=2.126550276 podStartE2EDuration="2.126550276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:50.124345779 +0000 UTC m=+2604.288898318" watchObservedRunningTime="2026-11-02 15:40:50.126550276 +0000 UTC m=+2604.291102816" Nov 02 15:40:50.946097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:50.946058 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:50.952560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:50.952531 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:50.975301 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:50.975273 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:50.976132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:50.976112 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:40:51.513335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.513297 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f"] Nov 02 15:40:51.517851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.517831 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.569478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.568784 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:40:51.569478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.568927 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:40:51.571559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.571531 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f"] Nov 02 15:40:51.632548 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.632497 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.632727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.632627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.632727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.632687 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhzn8\" (UniqueName: \"kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.733489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.733458 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.733650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.733510 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.733650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.733572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhzn8\" (UniqueName: \"kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.734158 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.734132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.736004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.735979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.792638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.792572 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhzn8\" (UniqueName: \"kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8\") pod \"maas-api-revoke-keys-e2e-worker-w5-891723-jm67f\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:51.831648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:51.831618 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:52.210909 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:52.210877 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f"] Nov 02 15:40:52.214825 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:40:52.214791 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaec6292_1884_43eb_a8cc_8c70fe9e84bb.slice/crio-cbe6f739a197ee8b4cd2df76c07fc0058e94fc943318d21574b03c20337f23e6 WatchSource:0}: Error finding container cbe6f739a197ee8b4cd2df76c07fc0058e94fc943318d21574b03c20337f23e6: Status 404 returned error can't find the container with id cbe6f739a197ee8b4cd2df76c07fc0058e94fc943318d21574b03c20337f23e6 Nov 02 15:40:52.990404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:52.990369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" event={"ID":"aaec6292-1884-43eb-a8cc-8c70fe9e84bb","Type":"ContainerStarted","Data":"9769685228b47c0d7dfd2ad0d6b5f079a347c4c4f0a39f4a14f550716be90c42"} Nov 02 15:40:52.990641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:52.990424 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" event={"ID":"aaec6292-1884-43eb-a8cc-8c70fe9e84bb","Type":"ContainerStarted","Data":"cbe6f739a197ee8b4cd2df76c07fc0058e94fc943318d21574b03c20337f23e6"} Nov 02 15:40:53.103649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:53.103595 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" podStartSLOduration=2.103577701 podStartE2EDuration="2.103577701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:53.101680509 +0000 UTC m=+2607.266233050" watchObservedRunningTime="2026-11-02 15:40:53.103577701 +0000 UTC m=+2607.268130242" Nov 02 15:40:54.000015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:53.999966 5091 generic.go:356] "Generic (PLEG): container finished" podID="aaec6292-1884-43eb-a8cc-8c70fe9e84bb" containerID="9769685228b47c0d7dfd2ad0d6b5f079a347c4c4f0a39f4a14f550716be90c42" exitCode=0 Nov 02 15:40:54.000494 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:54.000062 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" event={"ID":"aaec6292-1884-43eb-a8cc-8c70fe9e84bb","Type":"ContainerDied","Data":"9769685228b47c0d7dfd2ad0d6b5f079a347c4c4f0a39f4a14f550716be90c42"} Nov 02 15:40:55.160309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.160282 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:55.273770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.273688 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca\") pod \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " Nov 02 15:40:55.273941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.273827 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca\") pod \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " Nov 02 15:40:55.274011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.273964 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8\" (UniqueName: \"kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8\") pod \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\" (UID: \"aaec6292-1884-43eb-a8cc-8c70fe9e84bb\") " Nov 02 15:40:55.274094 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.274064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca" pod "aaec6292-1884-43eb-a8cc-8c70fe9e84bb" (UID: "aaec6292-1884-43eb-a8cc-8c70fe9e84bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:40:55.274454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.274432 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:55.276506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.276478 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca" pod "aaec6292-1884-43eb-a8cc-8c70fe9e84bb" (UID: "aaec6292-1884-43eb-a8cc-8c70fe9e84bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:40:55.276731 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.276711 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8" pod "aaec6292-1884-43eb-a8cc-8c70fe9e84bb" (UID: "aaec6292-1884-43eb-a8cc-8c70fe9e84bb"). InnerVolumeSpecName "kube-api-access-nhzn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:40:55.375828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.375797 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:55.375828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:55.375825 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhzn8\" (UniqueName: \"kubernetes.io/projected/aaec6292-1884-43eb-a8cc-8c70fe9e84bb-kube-api-access-nhzn8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:40:56.043121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:56.043090 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" Nov 02 15:40:56.044848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:56.044811 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f" event={"ID":"aaec6292-1884-43eb-a8cc-8c70fe9e84bb","Type":"ContainerDied","Data":"cbe6f739a197ee8b4cd2df76c07fc0058e94fc943318d21574b03c20337f23e6"} Nov 02 15:40:56.044986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:56.044871 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbe6f739a197ee8b4cd2df76c07fc0058e94fc943318d21574b03c20337f23e6" Nov 02 15:40:56.044986 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:56.044887 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl"] Nov 02 15:40:57.090051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.090016 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9"] Nov 02 15:40:57.090643 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.090628 5091 cpu_manager.go:414] "containerMap: removing container" podUID="aaec6292-1884-43eb-a8cc-8c70fe9e84bb" containerName="revoke-keys" Nov 02 15:40:57.090693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.090643 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaec6292-1884-43eb-a8cc-8c70fe9e84bb" containerName="revoke-keys" Nov 02 15:40:57.090755 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.090744 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaec6292-1884-43eb-a8cc-8c70fe9e84bb" containerName="revoke-keys" Nov 02 15:40:57.094370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.094354 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.149647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.149616 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9"] Nov 02 15:40:57.191358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.191328 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b1ee37-dockercfg-sj8st\"" Nov 02 15:40:57.191358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.191328 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b1ee37\"" Nov 02 15:40:57.217898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.217869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.218016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.217918 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2q4\" (UniqueName: \"kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4\") pod \"payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.318931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.318895 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.319068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.318950 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2q4\" (UniqueName: \"kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4\") pod \"payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.319668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.319650 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.377585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.377555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2q4\" (UniqueName: \"kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4\") pod \"payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.407671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.407644 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:57.796572 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:57.796480 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9"] Nov 02 15:40:57.800689 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:40:57.800664 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd9f591_c603_423f_a915_cbf7cd4aabaf.slice/crio-a518aa047a0225d804b4f96a5892defa522b727c1220168a02308c01d7797d3a WatchSource:0}: Error finding container a518aa047a0225d804b4f96a5892defa522b727c1220168a02308c01d7797d3a: Status 404 returned error can't find the container with id a518aa047a0225d804b4f96a5892defa522b727c1220168a02308c01d7797d3a Nov 02 15:40:57.936551 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:40:57.936055 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:40:58.062414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:58.062313 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" event={"ID":"7cd9f591-c603-423f-a915-cbf7cd4aabaf","Type":"ContainerStarted","Data":"da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f"} Nov 02 15:40:58.062414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:58.062382 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" event={"ID":"7cd9f591-c603-423f-a915-cbf7cd4aabaf","Type":"ContainerStarted","Data":"a518aa047a0225d804b4f96a5892defa522b727c1220168a02308c01d7797d3a"} Nov 02 15:40:58.062414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:58.062411 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:40:58.176120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:40:58.176070 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" podStartSLOduration=1.176057002 podStartE2EDuration="1.176057002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:40:58.172925377 +0000 UTC m=+2612.337477918" watchObservedRunningTime="2026-11-02 15:40:58.176057002 +0000 UTC m=+2612.340609542" Nov 02 15:41:02.048279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:02.048240 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz"] Nov 02 15:41:02.157316 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:02.157172 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72"] Nov 02 15:41:05.412270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.412237 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9"] Nov 02 15:41:05.416371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.416351 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.452299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.452268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.452626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.452599 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.452861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.452804 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.453035 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.453017 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.453227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.453208 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.453406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.453385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndnzl\" (UniqueName: \"kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.518029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.517996 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2203c8\"/\"default-dockercfg-v6n74\"" Nov 02 15:41:05.524878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.524795 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9"] Nov 02 15:41:05.525124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.525110 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2203c8\"/\"kube-root-ca.crt\"" Nov 02 15:41:05.525811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.525773 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2203c8\"/\"openshift-service-ca.crt\"" Nov 02 15:41:05.529867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.529436 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2203c8\"/\"test-model-2203c8-kserve-self-signed-certs\"" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.558433 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndnzl\" (UniqueName: \"kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.558567 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.558634 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.558659 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.558727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.558822 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.574371 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.584036 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.587807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.587754 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.658762 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.625980 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.660478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.660410 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.665542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.665461 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndnzl\" (UniqueName: \"kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl\") pod \"test-model-2203c8-kserve-d6b4c787-s4md9\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:05.736186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:05.735962 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:06.165019 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:06.164985 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9"] Nov 02 15:41:06.169038 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:41:06.169001 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b6ae906_0e4b_45d5_82d5_79a1f42d8d93.slice/crio-1493a2b24bc31a596f9755edcb7cd97eb3d2e38dfe926130fd1a5bcc247ffb66 WatchSource:0}: Error finding container 1493a2b24bc31a596f9755edcb7cd97eb3d2e38dfe926130fd1a5bcc247ffb66: Status 404 returned error can't find the container with id 1493a2b24bc31a596f9755edcb7cd97eb3d2e38dfe926130fd1a5bcc247ffb66 Nov 02 15:41:07.048809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:07.048765 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerName="payload-processing" containerID="cri-o://785fbe68559036c4601f16d303c13d38091ecb6dd547959de1ae9c3f0d28d0d9" gracePeriod=25 Nov 02 15:41:07.139588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:07.139544 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" event={"ID":"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93","Type":"ContainerStarted","Data":"eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616"} Nov 02 15:41:07.139588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:07.139589 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" event={"ID":"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93","Type":"ContainerStarted","Data":"1493a2b24bc31a596f9755edcb7cd97eb3d2e38dfe926130fd1a5bcc247ffb66"} Nov 02 15:41:07.158317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:07.158265 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerName="payload-pre-processing" containerID="cri-o://f119362dd89f91f8ccfa8921b2659550d267c9b82ec625494a6814124d42f843" gracePeriod=25 Nov 02 15:41:07.254968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:07.254906 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" podStartSLOduration=2.254886388 podStartE2EDuration="2.254886388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:41:07.253670544 +0000 UTC m=+2621.418223092" watchObservedRunningTime="2026-11-02 15:41:07.254886388 +0000 UTC m=+2621.419438929" Nov 02 15:41:07.737185 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:07.737149 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:09.073441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:09.073404 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:41:09.150790 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:09.150754 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:41:09.150932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:09.150792 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:41:09.409915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:09.409877 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz"] Nov 02 15:41:09.412923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:09.412900 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-61a835-wc6cz"] Nov 02 15:41:09.960848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:09.960814 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c6cdfb-e580-44c5-85ff-b50ab85454b0" path="/var/lib/kubelet/pods/d2c6cdfb-e580-44c5-85ff-b50ab85454b0/volumes" Nov 02 15:41:11.069821 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.069775 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" containerID="cri-o://fef7dc727d057df252ebb8e6a31bba3d4258069c098dfcf2f58a37d5d9ae5364" gracePeriod=45 Nov 02 15:41:11.189707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.189673 5091 generic.go:356] "Generic (PLEG): container finished" podID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerID="fef7dc727d057df252ebb8e6a31bba3d4258069c098dfcf2f58a37d5d9ae5364" exitCode=0 Nov 02 15:41:11.189829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.189750 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" event={"ID":"c4653ab7-34d7-4687-9a0f-9a194ba66cdb","Type":"ContainerDied","Data":"fef7dc727d057df252ebb8e6a31bba3d4258069c098dfcf2f58a37d5d9ae5364"} Nov 02 15:41:11.345125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.345096 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:41:11.451844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.451811 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache\") pod \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " Nov 02 15:41:11.452012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.451917 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home\") pod \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " Nov 02 15:41:11.452012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452005 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm\") pod \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " Nov 02 15:41:11.452112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452067 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir\") pod \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " Nov 02 15:41:11.452194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452179 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx\" (UniqueName: \"kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx\") pod \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " Nov 02 15:41:11.452242 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452220 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs\") pod \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\" (UID: \"c4653ab7-34d7-4687-9a0f-9a194ba66cdb\") " Nov 02 15:41:11.452457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir" pod "c4653ab7-34d7-4687-9a0f-9a194ba66cdb" (UID: "c4653ab7-34d7-4687-9a0f-9a194ba66cdb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.452599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452560 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home" pod "c4653ab7-34d7-4687-9a0f-9a194ba66cdb" (UID: "c4653ab7-34d7-4687-9a0f-9a194ba66cdb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.452793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452770 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.452903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452796 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.452903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.452853 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache" pod "c4653ab7-34d7-4687-9a0f-9a194ba66cdb" (UID: "c4653ab7-34d7-4687-9a0f-9a194ba66cdb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.462747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.461213 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs" pod "c4653ab7-34d7-4687-9a0f-9a194ba66cdb" (UID: "c4653ab7-34d7-4687-9a0f-9a194ba66cdb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:41:11.462747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.461243 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx" pod "c4653ab7-34d7-4687-9a0f-9a194ba66cdb" (UID: "c4653ab7-34d7-4687-9a0f-9a194ba66cdb"). InnerVolumeSpecName "kube-api-access-5qlbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:11.462747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.461283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm" pod "c4653ab7-34d7-4687-9a0f-9a194ba66cdb" (UID: "c4653ab7-34d7-4687-9a0f-9a194ba66cdb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:11.554132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.554097 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.554132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.554134 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qlbx\" (UniqueName: \"kubernetes.io/projected/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-kube-api-access-5qlbx\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.554305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.554152 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:11.554305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:11.554167 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4653ab7-34d7-4687-9a0f-9a194ba66cdb-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:12.207293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:12.207234 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" event={"ID":"c4653ab7-34d7-4687-9a0f-9a194ba66cdb","Type":"ContainerDied","Data":"d0d67fc7a0245da4263485953416e14eb00da618ebf6f0e39b4b214ba8c6fd98"} Nov 02 15:41:12.207765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:12.207341 5091 scope.go:122] "RemoveContainer" containerID="fef7dc727d057df252ebb8e6a31bba3d4258069c098dfcf2f58a37d5d9ae5364" Nov 02 15:41:12.207765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:12.207492 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl" Nov 02 15:41:12.368671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:12.368636 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl"] Nov 02 15:41:12.373763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:12.373737 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-891723/simulator-w5-891723-kserve-755b4d8fd4-ltthl"] Nov 02 15:41:13.961063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:13.961030 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" path="/var/lib/kubelet/pods/c4653ab7-34d7-4687-9a0f-9a194ba66cdb/volumes" Nov 02 15:41:15.736879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:15.736843 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:15.750783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:15.750760 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:16.218658 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:16.218623 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb"] Nov 02 15:41:16.218859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:16.218837 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" containerID="cri-o://1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949" gracePeriod=30 Nov 02 15:41:16.247619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:16.247591 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:41:19.150949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:19.150910 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:41:19.153640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:19.153608 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:41:21.486964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.486904 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:41:21.575712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575675 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.575912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575739 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.575912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575796 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.575912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575852 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.575912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575879 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.576119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.576119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.575995 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.576119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576043 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.576119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576087 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data\") pod \"68c74967-4c36-4996-ba34-099e94f3f53a\" (UID: \"68c74967-4c36-4996-ba34-099e94f3f53a\") " Nov 02 15:41:21.576325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576105 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.576325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576198 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.576431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576340 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.576485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576457 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:41:21.576625 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576598 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.576801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576785 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.576847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576804 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.576847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576813 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.576847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576823 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68c74967-4c36-4996-ba34-099e94f3f53a-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.576847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.576831 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.578532 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.578499 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:41:21.578653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.578571 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "kube-api-access-wwr8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:21.578653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.578587 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:41:21.578653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.578642 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token" pod "68c74967-4c36-4996-ba34-099e94f3f53a" (UID: "68c74967-4c36-4996-ba34-099e94f3f53a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:21.678290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.678259 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwr8p\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-kube-api-access-wwr8p\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.678290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.678290 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68c74967-4c36-4996-ba34-099e94f3f53a-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.678504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.678299 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68c74967-4c36-4996-ba34-099e94f3f53a-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:21.678504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:21.678308 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68c74967-4c36-4996-ba34-099e94f3f53a-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:22.283579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.283483 5091 generic.go:356] "Generic (PLEG): container finished" podID="68c74967-4c36-4996-ba34-099e94f3f53a" containerID="1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949" exitCode=0 Nov 02 15:41:22.283579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.283561 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" Nov 02 15:41:22.283761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.283576 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" event={"ID":"68c74967-4c36-4996-ba34-099e94f3f53a","Type":"ContainerDied","Data":"1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949"} Nov 02 15:41:22.283761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.283642 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb" event={"ID":"68c74967-4c36-4996-ba34-099e94f3f53a","Type":"ContainerDied","Data":"dda312f6b0fa0487475c04c922f4a91c389fcb6609e467a6e56fa29e4fa264f2"} Nov 02 15:41:22.283761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.283668 5091 scope.go:122] "RemoveContainer" containerID="1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949" Nov 02 15:41:22.294406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.294389 5091 scope.go:122] "RemoveContainer" containerID="1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949" Nov 02 15:41:22.294689 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:41:22.294670 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949\": container with ID starting with 1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949 not found: ID does not exist" containerID="1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949" Nov 02 15:41:22.294779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.294695 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949"} err="failed to get container status \"1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949\": rpc error: code = NotFound desc = could not find container \"1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949\": container with ID starting with 1ce65309d5d55786c8401d00ece097da00a1aeee30e4d85414cfb28d230cb949 not found: ID does not exist" Nov 02 15:41:22.490147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.490115 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb"] Nov 02 15:41:22.496682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:22.496655 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-61a835-openshift-default-7f845459c-mw2fb"] Nov 02 15:41:23.961028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:23.960995 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" path="/var/lib/kubelet/pods/68c74967-4c36-4996-ba34-099e94f3f53a/volumes" Nov 02 15:41:27.318285 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.318175 5091 generic.go:356] "Generic (PLEG): container finished" podID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerID="f119362dd89f91f8ccfa8921b2659550d267c9b82ec625494a6814124d42f843" exitCode=0 Nov 02 15:41:27.318684 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.318346 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" event={"ID":"a08a7a22-5b7e-48b9-9fa2-33ea61732157","Type":"ContainerDied","Data":"f119362dd89f91f8ccfa8921b2659550d267c9b82ec625494a6814124d42f843"} Nov 02 15:41:27.319768 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.319743 5091 generic.go:356] "Generic (PLEG): container finished" podID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerID="785fbe68559036c4601f16d303c13d38091ecb6dd547959de1ae9c3f0d28d0d9" exitCode=0 Nov 02 15:41:27.319900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.319854 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" event={"ID":"96bc8d73-120a-4019-a9fd-41d3265d7a39","Type":"ContainerDied","Data":"785fbe68559036c4601f16d303c13d38091ecb6dd547959de1ae9c3f0d28d0d9"} Nov 02 15:41:27.319900 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.319889 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" event={"ID":"96bc8d73-120a-4019-a9fd-41d3265d7a39","Type":"ContainerDied","Data":"d4632fc5c4626185d09015284b43a743e2720e9e4a0b676075b4362032929fe4"} Nov 02 15:41:27.320029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.319906 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4632fc5c4626185d09015284b43a743e2720e9e4a0b676075b4362032929fe4" Nov 02 15:41:27.337584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.336294 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:41:27.433134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.433108 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:41:27.458809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.458774 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x\" (UniqueName: \"kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x\") pod \"96bc8d73-120a-4019-a9fd-41d3265d7a39\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " Nov 02 15:41:27.458974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.458823 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume\") pod \"96bc8d73-120a-4019-a9fd-41d3265d7a39\" (UID: \"96bc8d73-120a-4019-a9fd-41d3265d7a39\") " Nov 02 15:41:27.459636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.459600 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume" pod "96bc8d73-120a-4019-a9fd-41d3265d7a39" (UID: "96bc8d73-120a-4019-a9fd-41d3265d7a39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:41:27.461428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.461403 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x" pod "96bc8d73-120a-4019-a9fd-41d3265d7a39" (UID: "96bc8d73-120a-4019-a9fd-41d3265d7a39"). InnerVolumeSpecName "kube-api-access-fdt5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:27.559901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.559856 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq\" (UniqueName: \"kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq\") pod \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " Nov 02 15:41:27.560095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.559923 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume\") pod \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\" (UID: \"a08a7a22-5b7e-48b9-9fa2-33ea61732157\") " Nov 02 15:41:27.563739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.560579 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdt5x\" (UniqueName: \"kubernetes.io/projected/96bc8d73-120a-4019-a9fd-41d3265d7a39-kube-api-access-fdt5x\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:27.563739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.560605 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96bc8d73-120a-4019-a9fd-41d3265d7a39-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:27.563739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.560957 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume" pod "a08a7a22-5b7e-48b9-9fa2-33ea61732157" (UID: "a08a7a22-5b7e-48b9-9fa2-33ea61732157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:41:27.564095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.564066 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq" pod "a08a7a22-5b7e-48b9-9fa2-33ea61732157" (UID: "a08a7a22-5b7e-48b9-9fa2-33ea61732157"). InnerVolumeSpecName "kube-api-access-64qmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:41:27.661765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.661729 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qmq\" (UniqueName: \"kubernetes.io/projected/a08a7a22-5b7e-48b9-9fa2-33ea61732157-kube-api-access-64qmq\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:27.661765 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:27.661757 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08a7a22-5b7e-48b9-9fa2-33ea61732157-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:41:27.937427 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:41:27.937380 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:41:28.327770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.327688 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" Nov 02 15:41:28.327770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.327752 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72" event={"ID":"a08a7a22-5b7e-48b9-9fa2-33ea61732157","Type":"ContainerDied","Data":"fc0c8880e6d7f41f83fdc1c8e68d58ba2e4c21b3875acdde70341c805e946a72"} Nov 02 15:41:28.328282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.327808 5091 scope.go:122] "RemoveContainer" containerID="f119362dd89f91f8ccfa8921b2659550d267c9b82ec625494a6814124d42f843" Nov 02 15:41:28.328282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.327887 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz" Nov 02 15:41:28.536399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.536363 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72"] Nov 02 15:41:28.583596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.583496 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-891723-55684b9bff-rlf72"] Nov 02 15:41:28.644241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.644199 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz"] Nov 02 15:41:28.647229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:28.647201 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-891723-875d6cb8f-7fvmz"] Nov 02 15:41:29.842694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:29.842662 5091 scope.go:122] "RemoveContainer" containerID="785fbe68559036c4601f16d303c13d38091ecb6dd547959de1ae9c3f0d28d0d9" Nov 02 15:41:29.960478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:29.960443 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" path="/var/lib/kubelet/pods/96bc8d73-120a-4019-a9fd-41d3265d7a39/volumes" Nov 02 15:41:29.960905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:29.960887 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" path="/var/lib/kubelet/pods/a08a7a22-5b7e-48b9-9fa2-33ea61732157/volumes" Nov 02 15:41:56.447594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.445889 5091 patch_prober.go:29] interesting pod/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": read tcp 10.132.0.2:47026->10.132.0.171:15021: read: connection reset by peer" start-of-body= Nov 02 15:41:56.447594 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.445947 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": read tcp 10.132.0.2:47026->10.132.0.171:15021: read: connection reset by peer" Nov 02 15:41:56.569192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.569159 5091 generic.go:356] "Generic (PLEG): container finished" podID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerID="55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62" exitCode=0 Nov 02 15:41:56.569369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.569237 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" event={"ID":"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1","Type":"ContainerDied","Data":"55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62"} Nov 02 15:41:56.569779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.569761 5091 scope.go:122] "RemoveContainer" containerID="55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62" Nov 02 15:41:56.571473 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.571443 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:41:56.571620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.571565 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:41:56.571690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:56.571632 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:41:57.579224 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:57.579187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" event={"ID":"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1","Type":"ContainerStarted","Data":"cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49"} Nov 02 15:41:57.957601 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:41:57.957563 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:41:58.289959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:58.289877 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:41:58.290138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:58.290074 5091 patch_prober.go:29] interesting pod/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 15:41:58.290213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:58.290125 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 15:41:59.290269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:59.290237 5091 patch_prober.go:29] interesting pod/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 15:41:59.290676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:41:59.290293 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 15:42:00.289705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.289671 5091 patch_prober.go:29] interesting pod/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 15:42:00.289891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.289737 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 15:42:00.365864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.365825 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd"] Nov 02 15:42:00.366465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366446 5091 cpu_manager.go:414] "containerMap: removing container" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" Nov 02 15:42:00.366531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366467 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" Nov 02 15:42:00.366531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366478 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerName="payload-pre-processing" Nov 02 15:42:00.366531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366484 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerName="payload-pre-processing" Nov 02 15:42:00.366531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366496 5091 cpu_manager.go:414] "containerMap: removing container" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerName="payload-processing" Nov 02 15:42:00.366531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366501 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerName="payload-processing" Nov 02 15:42:00.366531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366531 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" Nov 02 15:42:00.366746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366538 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" Nov 02 15:42:00.366746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366655 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c74967-4c36-4996-ba34-099e94f3f53a" containerName="istio-proxy" Nov 02 15:42:00.366746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366663 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08a7a22-5b7e-48b9-9fa2-33ea61732157" containerName="payload-pre-processing" Nov 02 15:42:00.366746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366670 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="96bc8d73-120a-4019-a9fd-41d3265d7a39" containerName="payload-processing" Nov 02 15:42:00.366746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.366676 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4653ab7-34d7-4687-9a0f-9a194ba66cdb" containerName="main" Nov 02 15:42:00.370325 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.370298 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.400630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.400597 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqds\" (UniqueName: \"kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.400778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.400747 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.400837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.400820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.400882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.400854 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.401012 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.400991 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.401093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.401035 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.470577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.468612 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f6b245\"/\"default-dockercfg-vcfpq\"" Nov 02 15:42:00.470577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.468863 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f6b245\"/\"kube-root-ca.crt\"" Nov 02 15:42:00.470577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.468935 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f6b245\"/\"test-model-f6b245-kserve-self-signed-certs\"" Nov 02 15:42:00.470577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.469023 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f6b245\"/\"openshift-service-ca.crt\"" Nov 02 15:42:00.475939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.475915 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd"] Nov 02 15:42:00.502567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.501991 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.502567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.502050 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.502567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.502129 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqds\" (UniqueName: \"kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.502567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.502215 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.502567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.502273 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.502567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.502300 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.503060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.502903 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.509936 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.503392 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.509936 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.503862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.510194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.510024 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.510981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.510956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.565393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.565312 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqds\" (UniqueName: \"kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds\") pod \"test-model-f6b245-kserve-56dbb8ddd9-99pzd\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:00.683360 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:00.683330 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:01.107029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.106985 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd"] Nov 02 15:42:01.111681 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:42:01.111656 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa350c5f_bf5c_4998_af63_41ab18d69f72.slice/crio-49fc1ca4e097c27a10a9caa48d21eaf57e194aae716a8ccef130147edf9106e2 WatchSource:0}: Error finding container 49fc1ca4e097c27a10a9caa48d21eaf57e194aae716a8ccef130147edf9106e2: Status 404 returned error can't find the container with id 49fc1ca4e097c27a10a9caa48d21eaf57e194aae716a8ccef130147edf9106e2 Nov 02 15:42:01.290128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.290092 5091 patch_prober.go:29] interesting pod/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 15:42:01.290296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.290160 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 15:42:01.607359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.607318 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" event={"ID":"fa350c5f-bf5c-4998-af63-41ab18d69f72","Type":"ContainerStarted","Data":"f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20"} Nov 02 15:42:01.607359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.607365 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" event={"ID":"fa350c5f-bf5c-4998-af63-41ab18d69f72","Type":"ContainerStarted","Data":"49fc1ca4e097c27a10a9caa48d21eaf57e194aae716a8ccef130147edf9106e2"} Nov 02 15:42:01.683909 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.683878 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:01.724693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:01.724635 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" podStartSLOduration=1.724616932 podStartE2EDuration="1.724616932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:01.719667259 +0000 UTC m=+2675.884219816" watchObservedRunningTime="2026-11-02 15:42:01.724616932 +0000 UTC m=+2675.889169474" Nov 02 15:42:02.290442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:02.290405 5091 patch_prober.go:29] interesting pod/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 02 15:42:02.290655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:02.290467 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 02 15:42:03.294483 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:03.294449 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:42:03.296558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:03.296495 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:42:03.304664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:03.304640 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:42:05.382279 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.382240 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs"] Nov 02 15:42:05.386582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.386560 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.431419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.431379 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq"] Nov 02 15:42:05.436471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.436448 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.473931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.473895 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:42:05.474163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.474062 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:42:05.479134 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.479110 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs"] Nov 02 15:42:05.480068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.480047 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.480155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.480097 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.480210 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.480190 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.480324 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.480302 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clcpm\" (UniqueName: \"kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.480452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.480438 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.480564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.480505 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7f2g\" (UniqueName: \"kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.489652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.489623 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq"] Nov 02 15:42:05.581689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.581655 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.581901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.581719 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clcpm\" (UniqueName: \"kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.581901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.581790 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.581901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.581853 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7f2g\" (UniqueName: \"kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.581901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.581888 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.582133 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.581942 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.582459 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.582427 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.582662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.582644 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.584219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.584195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.584323 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.584301 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.640377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.640300 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clcpm\" (UniqueName: \"kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm\") pod \"maas-api-revoke-sub-22b2afe9-4zjnq\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.640530 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.640390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7f2g\" (UniqueName: \"kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g\") pod \"maas-api-revoke-sub-f121d918-sc9hs\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.701191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.701159 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:05.755967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.755930 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:05.902726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.902645 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6"] Nov 02 15:42:05.908863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.908841 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:05.965109 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.965061 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b1ee37-serving-cert\"" Nov 02 15:42:05.973744 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.973710 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6"] Nov 02 15:42:05.990192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.989749 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:05.990192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.989797 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:05.990192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.989846 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:05.990192 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:05.989927 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbl59\" (UniqueName: \"kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.090921 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.090888 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbl59\" (UniqueName: \"kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.091110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.091052 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.091110 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.091084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.091243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.091114 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.091243 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:42:06.091235 5091 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-serving-cert: secret "maas-api-e2e-api-b-b1ee37-serving-cert" not found Nov 02 15:42:06.091341 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:42:06.091302 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls podName:b47059a3-a218-4d1c-9e27-5d4a5d968ecc nodeName:}" failed. No retries permitted until 2026-11-02 15:42:06.591283136 +0000 UTC m=+2680.755835653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls") pod "maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" (UID: "b47059a3-a218-4d1c-9e27-5d4a5d968ecc") : secret "maas-api-e2e-api-b-b1ee37-serving-cert" not found Nov 02 15:42:06.091855 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.091831 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.093627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.093602 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.109988 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.109956 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs"] Nov 02 15:42:06.132297 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:42:06.132269 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd174e335_7f8d_488b_98e6_36a8f033417a.slice/crio-7a5a919245aa0d2f7b1d589458561df6047749ae5d29a26512c19c14218d5da0 WatchSource:0}: Error finding container 7a5a919245aa0d2f7b1d589458561df6047749ae5d29a26512c19c14218d5da0: Status 404 returned error can't find the container with id 7a5a919245aa0d2f7b1d589458561df6047749ae5d29a26512c19c14218d5da0 Nov 02 15:42:06.150405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.150372 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbl59\" (UniqueName: \"kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.162917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.162856 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq"] Nov 02 15:42:06.166977 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:42:06.166946 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b33bc4_9dea_4e40_874a_955223c7c750.slice/crio-2f59f7fd0776efd1169fc212052a2f7ccb07f5389608bfa3759bc5406abe9c0e WatchSource:0}: Error finding container 2f59f7fd0776efd1169fc212052a2f7ccb07f5389608bfa3759bc5406abe9c0e: Status 404 returned error can't find the container with id 2f59f7fd0776efd1169fc212052a2f7ccb07f5389608bfa3759bc5406abe9c0e Nov 02 15:42:06.597848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.597815 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.600447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.600411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\") pod \"maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.647188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.647132 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" event={"ID":"55b33bc4-9dea-4e40-874a-955223c7c750","Type":"ContainerStarted","Data":"504dcefd932d6db7fdaf72553ad0a38b4fc78d143ccb085f99b628bdacf47b0c"} Nov 02 15:42:06.647381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.647209 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" event={"ID":"55b33bc4-9dea-4e40-874a-955223c7c750","Type":"ContainerStarted","Data":"2f59f7fd0776efd1169fc212052a2f7ccb07f5389608bfa3759bc5406abe9c0e"} Nov 02 15:42:06.648965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.648930 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" event={"ID":"d174e335-7f8d-488b-98e6-36a8f033417a","Type":"ContainerStarted","Data":"595b6a5f070f844c78442bd9e5b4f29d872d0b5b2812de120724eae81ca11e6a"} Nov 02 15:42:06.649112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.648981 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" event={"ID":"d174e335-7f8d-488b-98e6-36a8f033417a","Type":"ContainerStarted","Data":"7a5a919245aa0d2f7b1d589458561df6047749ae5d29a26512c19c14218d5da0"} Nov 02 15:42:06.761571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.761439 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" podStartSLOduration=1.7614247870000002 podStartE2EDuration="1.761424787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:06.757632585 +0000 UTC m=+2680.922185126" watchObservedRunningTime="2026-11-02 15:42:06.761424787 +0000 UTC m=+2680.925977327" Nov 02 15:42:06.822723 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.822694 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:06.885199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:06.885148 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" podStartSLOduration=1.8851336760000001 podStartE2EDuration="1.885133676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:06.880077889 +0000 UTC m=+2681.044630429" watchObservedRunningTime="2026-11-02 15:42:06.885133676 +0000 UTC m=+2681.049686213" Nov 02 15:42:07.218940 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.218904 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6"] Nov 02 15:42:07.223510 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:42:07.223477 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47059a3_a218_4d1c_9e27_5d4a5d968ecc.slice/crio-1cfb73cc9f965f70152a72664208b3e2224789c6acd82cf9cdd75cd4c688b3ac WatchSource:0}: Error finding container 1cfb73cc9f965f70152a72664208b3e2224789c6acd82cf9cdd75cd4c688b3ac: Status 404 returned error can't find the container with id 1cfb73cc9f965f70152a72664208b3e2224789c6acd82cf9cdd75cd4c688b3ac Nov 02 15:42:07.225097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.225076 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 15:42:07.659759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.659717 5091 generic.go:356] "Generic (PLEG): container finished" podID="55b33bc4-9dea-4e40-874a-955223c7c750" containerID="504dcefd932d6db7fdaf72553ad0a38b4fc78d143ccb085f99b628bdacf47b0c" exitCode=0 Nov 02 15:42:07.660189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.659804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" event={"ID":"55b33bc4-9dea-4e40-874a-955223c7c750","Type":"ContainerDied","Data":"504dcefd932d6db7fdaf72553ad0a38b4fc78d143ccb085f99b628bdacf47b0c"} Nov 02 15:42:07.661807 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.661781 5091 generic.go:356] "Generic (PLEG): container finished" podID="d174e335-7f8d-488b-98e6-36a8f033417a" containerID="595b6a5f070f844c78442bd9e5b4f29d872d0b5b2812de120724eae81ca11e6a" exitCode=0 Nov 02 15:42:07.661939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.661817 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" event={"ID":"d174e335-7f8d-488b-98e6-36a8f033417a","Type":"ContainerDied","Data":"595b6a5f070f844c78442bd9e5b4f29d872d0b5b2812de120724eae81ca11e6a"} Nov 02 15:42:07.663222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:07.663202 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" event={"ID":"b47059a3-a218-4d1c-9e27-5d4a5d968ecc","Type":"ContainerStarted","Data":"1cfb73cc9f965f70152a72664208b3e2224789c6acd82cf9cdd75cd4c688b3ac"} Nov 02 15:42:08.671311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.671263 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" event={"ID":"b47059a3-a218-4d1c-9e27-5d4a5d968ecc","Type":"ContainerStarted","Data":"b2e9b56ed92856517c773edbdfcd4fd2fbedc51995323e62844ae2a41732d8d6"} Nov 02 15:42:08.671811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.671457 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:08.882014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.881988 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:08.884092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.884073 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:08.933233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.926966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm\" (UniqueName: \"kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm\") pod \"55b33bc4-9dea-4e40-874a-955223c7c750\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " Nov 02 15:42:08.933233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.927018 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g\" (UniqueName: \"kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g\") pod \"d174e335-7f8d-488b-98e6-36a8f033417a\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " Nov 02 15:42:08.933233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.927078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca\") pod \"d174e335-7f8d-488b-98e6-36a8f033417a\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " Nov 02 15:42:08.933233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.927163 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca\") pod \"d174e335-7f8d-488b-98e6-36a8f033417a\" (UID: \"d174e335-7f8d-488b-98e6-36a8f033417a\") " Nov 02 15:42:08.933233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.927302 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca\") pod \"55b33bc4-9dea-4e40-874a-955223c7c750\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " Nov 02 15:42:08.933233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.927382 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca\") pod \"55b33bc4-9dea-4e40-874a-955223c7c750\" (UID: \"55b33bc4-9dea-4e40-874a-955223c7c750\") " Nov 02 15:42:08.957459 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.955430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca" pod "55b33bc4-9dea-4e40-874a-955223c7c750" (UID: "55b33bc4-9dea-4e40-874a-955223c7c750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:42:08.958073 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.958045 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca" pod "d174e335-7f8d-488b-98e6-36a8f033417a" (UID: "d174e335-7f8d-488b-98e6-36a8f033417a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:42:08.971581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.966938 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca" pod "55b33bc4-9dea-4e40-874a-955223c7c750" (UID: "55b33bc4-9dea-4e40-874a-955223c7c750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:42:08.971581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.966950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g" pod "d174e335-7f8d-488b-98e6-36a8f033417a" (UID: "d174e335-7f8d-488b-98e6-36a8f033417a"). InnerVolumeSpecName "kube-api-access-z7f2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:08.971581 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.968770 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm" pod "55b33bc4-9dea-4e40-874a-955223c7c750" (UID: "55b33bc4-9dea-4e40-874a-955223c7c750"). InnerVolumeSpecName "kube-api-access-clcpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:08.973724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:08.973687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca" pod "d174e335-7f8d-488b-98e6-36a8f033417a" (UID: "d174e335-7f8d-488b-98e6-36a8f033417a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:42:09.011545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.011149 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" podStartSLOduration=3.109517615 podStartE2EDuration="4.011122949s" totalImagesPullingTime="901.605334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:08.793815582 +0000 UTC m=+2682.958368125" watchObservedRunningTime="2026-11-02 15:42:09.011122949 +0000 UTC m=+2683.175675490" Nov 02 15:42:09.029560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.029497 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:09.029795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.029774 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b33bc4-9dea-4e40-874a-955223c7c750-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:09.029922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.029908 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clcpm\" (UniqueName: \"kubernetes.io/projected/55b33bc4-9dea-4e40-874a-955223c7c750-kube-api-access-clcpm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:09.030026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.030013 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7f2g\" (UniqueName: \"kubernetes.io/projected/d174e335-7f8d-488b-98e6-36a8f033417a-kube-api-access-z7f2g\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:09.030121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.030108 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:09.030215 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.030203 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d174e335-7f8d-488b-98e6-36a8f033417a-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:09.681438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.681405 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" Nov 02 15:42:09.681853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.681441 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq" event={"ID":"55b33bc4-9dea-4e40-874a-955223c7c750","Type":"ContainerDied","Data":"2f59f7fd0776efd1169fc212052a2f7ccb07f5389608bfa3759bc5406abe9c0e"} Nov 02 15:42:09.681853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.681495 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f59f7fd0776efd1169fc212052a2f7ccb07f5389608bfa3759bc5406abe9c0e" Nov 02 15:42:09.682932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.682909 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" Nov 02 15:42:09.683056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.682937 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs" event={"ID":"d174e335-7f8d-488b-98e6-36a8f033417a","Type":"ContainerDied","Data":"7a5a919245aa0d2f7b1d589458561df6047749ae5d29a26512c19c14218d5da0"} Nov 02 15:42:09.683056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:09.682981 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a5a919245aa0d2f7b1d589458561df6047749ae5d29a26512c19c14218d5da0" Nov 02 15:42:10.684588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:10.684554 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:10.699262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:10.699234 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:11.697380 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:11.697351 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:42:14.688495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:14.688465 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:42:27.934986 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:42:27.934942 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:42:28.767996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:28.767962 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:42:28.786617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:28.786586 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:42:29.898311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:29.898282 5091 scope.go:122] "RemoveContainer" containerID="ba4ac51bc7624c3f11ab332a6ed0c159453e9a2c315d8e938e758170f77bd1e7" Nov 02 15:42:43.701531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.701485 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp"] Nov 02 15:42:43.702119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.702101 5091 cpu_manager.go:414] "containerMap: removing container" podUID="55b33bc4-9dea-4e40-874a-955223c7c750" containerName="revoke-keys" Nov 02 15:42:43.702181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.702119 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b33bc4-9dea-4e40-874a-955223c7c750" containerName="revoke-keys" Nov 02 15:42:43.702181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.702147 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d174e335-7f8d-488b-98e6-36a8f033417a" containerName="revoke-keys" Nov 02 15:42:43.702181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.702152 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d174e335-7f8d-488b-98e6-36a8f033417a" containerName="revoke-keys" Nov 02 15:42:43.702344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.702232 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d174e335-7f8d-488b-98e6-36a8f033417a" containerName="revoke-keys" Nov 02 15:42:43.702344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.702245 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b33bc4-9dea-4e40-874a-955223c7c750" containerName="revoke-keys" Nov 02 15:42:43.707005 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.706987 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.761429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.761396 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp"] Nov 02 15:42:43.803350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.803322 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:42:43.803503 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.803416 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:42:43.804968 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.804931 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.805113 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.804992 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnhg\" (UniqueName: \"kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.805113 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.805089 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.906351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.906321 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.906530 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.906482 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.906579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.906553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnhg\" (UniqueName: \"kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.907093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.907069 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.908800 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.908777 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:43.965712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:43.965637 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnhg\" (UniqueName: \"kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg\") pod \"maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:44.020887 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:44.020853 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:44.424272 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:44.424235 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp"] Nov 02 15:42:44.456716 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:42:44.456677 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab79d46a_301c_43c8_ba7e_c1cfb09d820b.slice/crio-815e2b525a12cbe11c6f5a012cd259d0991534aecfda8b246178d3ebe84f0d49 WatchSource:0}: Error finding container 815e2b525a12cbe11c6f5a012cd259d0991534aecfda8b246178d3ebe84f0d49: Status 404 returned error can't find the container with id 815e2b525a12cbe11c6f5a012cd259d0991534aecfda8b246178d3ebe84f0d49 Nov 02 15:42:44.986299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:44.986263 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" event={"ID":"ab79d46a-301c-43c8-ba7e-c1cfb09d820b","Type":"ContainerStarted","Data":"b17a52397afa1c4f6c3789d1e3f4015a0009128d8c0c02eb7f54abb8d6d2c957"} Nov 02 15:42:44.986685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:44.986318 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" event={"ID":"ab79d46a-301c-43c8-ba7e-c1cfb09d820b","Type":"ContainerStarted","Data":"815e2b525a12cbe11c6f5a012cd259d0991534aecfda8b246178d3ebe84f0d49"} Nov 02 15:42:45.102240 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:45.102140 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" podStartSLOduration=2.102125274 podStartE2EDuration="2.102125274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:42:45.100918814 +0000 UTC m=+2719.265471356" watchObservedRunningTime="2026-11-02 15:42:45.102125274 +0000 UTC m=+2719.266677859" Nov 02 15:42:45.996382 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:45.996335 5091 generic.go:356] "Generic (PLEG): container finished" podID="ab79d46a-301c-43c8-ba7e-c1cfb09d820b" containerID="b17a52397afa1c4f6c3789d1e3f4015a0009128d8c0c02eb7f54abb8d6d2c957" exitCode=0 Nov 02 15:42:45.996816 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:45.996418 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" event={"ID":"ab79d46a-301c-43c8-ba7e-c1cfb09d820b","Type":"ContainerDied","Data":"b17a52397afa1c4f6c3789d1e3f4015a0009128d8c0c02eb7f54abb8d6d2c957"} Nov 02 15:42:47.162362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.162334 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:47.243753 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.243720 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca\") pod \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " Nov 02 15:42:47.243934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.243900 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca\") pod \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " Nov 02 15:42:47.244028 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.244010 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg\" (UniqueName: \"kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg\") pod \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\" (UID: \"ab79d46a-301c-43c8-ba7e-c1cfb09d820b\") " Nov 02 15:42:47.244232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.244134 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca" pod "ab79d46a-301c-43c8-ba7e-c1cfb09d820b" (UID: "ab79d46a-301c-43c8-ba7e-c1cfb09d820b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:42:47.244535 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.244489 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:47.246579 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.246509 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca" pod "ab79d46a-301c-43c8-ba7e-c1cfb09d820b" (UID: "ab79d46a-301c-43c8-ba7e-c1cfb09d820b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:42:47.246691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.246582 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg" pod "ab79d46a-301c-43c8-ba7e-c1cfb09d820b" (UID: "ab79d46a-301c-43c8-ba7e-c1cfb09d820b"). InnerVolumeSpecName "kube-api-access-cdnhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:42:47.345786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.345748 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:47.345786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:47.345787 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdnhg\" (UniqueName: \"kubernetes.io/projected/ab79d46a-301c-43c8-ba7e-c1cfb09d820b-kube-api-access-cdnhg\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:42:48.014016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:48.013983 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" Nov 02 15:42:48.014171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:48.014017 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp" event={"ID":"ab79d46a-301c-43c8-ba7e-c1cfb09d820b","Type":"ContainerDied","Data":"815e2b525a12cbe11c6f5a012cd259d0991534aecfda8b246178d3ebe84f0d49"} Nov 02 15:42:48.014171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:48.014066 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="815e2b525a12cbe11c6f5a012cd259d0991534aecfda8b246178d3ebe84f0d49" Nov 02 15:42:49.023644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:49.023551 5091 generic.go:356] "Generic (PLEG): container finished" podID="3763b422-89cf-4336-8097-cec4de9907bd" containerID="95e0798e801764ce46fcfc6205278c2b03fe90b877d062d7a914dc1faaf01314" exitCode=0 Nov 02 15:42:49.023644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:49.023628 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" event={"ID":"3763b422-89cf-4336-8097-cec4de9907bd","Type":"ContainerDied","Data":"95e0798e801764ce46fcfc6205278c2b03fe90b877d062d7a914dc1faaf01314"} Nov 02 15:42:49.024161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:49.024048 5091 scope.go:122] "RemoveContainer" containerID="95e0798e801764ce46fcfc6205278c2b03fe90b877d062d7a914dc1faaf01314" Nov 02 15:42:49.027914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:49.026222 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:42:49.027914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:49.026397 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:42:49.027914 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:49.026494 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:42:50.060542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:50.056216 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" event={"ID":"3763b422-89cf-4336-8097-cec4de9907bd","Type":"ContainerStarted","Data":"c4835ad114ef6275f17cdf8821ab5f891396f5f16c6cec5d43f7a6713105c7e1"} Nov 02 15:42:50.948540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:50.946654 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:42:50.948540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:50.946794 5091 patch_prober.go:29] interesting pod/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 15:42:50.948540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:50.946837 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 15:42:50.975920 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:50.975880 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:42:51.954540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:51.949901 5091 patch_prober.go:29] interesting pod/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 15:42:51.954540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:51.949968 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 15:42:52.949543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:52.948893 5091 patch_prober.go:29] interesting pod/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 15:42:52.949543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:52.948955 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 15:42:53.948963 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:53.948920 5091 patch_prober.go:29] interesting pod/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 15:42:53.949640 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:53.949602 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 15:42:54.952995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:54.952956 5091 patch_prober.go:29] interesting pod/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Nov 02 15:42:54.954622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:54.954588 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Nov 02 15:42:55.951022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:55.950990 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:42:55.959925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:42:55.959899 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:42:57.936459 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:42:57.936418 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:43:18.782869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.782784 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt"] Nov 02 15:43:18.783457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.783438 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ab79d46a-301c-43c8-ba7e-c1cfb09d820b" containerName="revoke-keys" Nov 02 15:43:18.783505 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.783458 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab79d46a-301c-43c8-ba7e-c1cfb09d820b" containerName="revoke-keys" Nov 02 15:43:18.783586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.783574 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab79d46a-301c-43c8-ba7e-c1cfb09d820b" containerName="revoke-keys" Nov 02 15:43:18.787308 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.787292 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.842053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.842019 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt"] Nov 02 15:43:18.876705 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.876650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkrm\" (UniqueName: \"kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.876902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.876809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.876902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.876870 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.883974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.883948 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:43:18.883974 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.883966 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:43:18.978485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.978453 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkrm\" (UniqueName: \"kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.978650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.978560 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.978650 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.978607 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.979249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.979227 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:18.981021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:18.980998 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:19.040555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:19.040451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkrm\" (UniqueName: \"kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm\") pod \"maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:19.103095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:19.103061 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:19.485370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:19.485341 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt"] Nov 02 15:43:19.488895 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:43:19.488865 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122a5c3c_d745_469e_8478_d83d6ab6f86a.slice/crio-f82f8675e72286a9c87528696c376750828f875151d0116a20c314be5c79524d WatchSource:0}: Error finding container f82f8675e72286a9c87528696c376750828f875151d0116a20c314be5c79524d: Status 404 returned error can't find the container with id f82f8675e72286a9c87528696c376750828f875151d0116a20c314be5c79524d Nov 02 15:43:20.279056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:20.279012 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" event={"ID":"122a5c3c-d745-469e-8478-d83d6ab6f86a","Type":"ContainerStarted","Data":"7d186786933b7272522da3b70e9fbbfe2e3d189a6efbcff6fa585696a70b4a10"} Nov 02 15:43:20.279543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:20.279082 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" event={"ID":"122a5c3c-d745-469e-8478-d83d6ab6f86a","Type":"ContainerStarted","Data":"f82f8675e72286a9c87528696c376750828f875151d0116a20c314be5c79524d"} Nov 02 15:43:20.433627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:20.433573 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" podStartSLOduration=2.433558246 podStartE2EDuration="2.433558246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:20.428853409 +0000 UTC m=+2754.593405949" watchObservedRunningTime="2026-11-02 15:43:20.433558246 +0000 UTC m=+2754.598110786" Nov 02 15:43:21.287200 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:21.287165 5091 generic.go:356] "Generic (PLEG): container finished" podID="122a5c3c-d745-469e-8478-d83d6ab6f86a" containerID="7d186786933b7272522da3b70e9fbbfe2e3d189a6efbcff6fa585696a70b4a10" exitCode=0 Nov 02 15:43:21.287642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:21.287248 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" event={"ID":"122a5c3c-d745-469e-8478-d83d6ab6f86a","Type":"ContainerDied","Data":"7d186786933b7272522da3b70e9fbbfe2e3d189a6efbcff6fa585696a70b4a10"} Nov 02 15:43:22.457504 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.457468 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:22.623867 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.623837 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm\" (UniqueName: \"kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm\") pod \"122a5c3c-d745-469e-8478-d83d6ab6f86a\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " Nov 02 15:43:22.624058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.623941 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca\") pod \"122a5c3c-d745-469e-8478-d83d6ab6f86a\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " Nov 02 15:43:22.624058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.623979 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca\") pod \"122a5c3c-d745-469e-8478-d83d6ab6f86a\" (UID: \"122a5c3c-d745-469e-8478-d83d6ab6f86a\") " Nov 02 15:43:22.624353 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.624328 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca" pod "122a5c3c-d745-469e-8478-d83d6ab6f86a" (UID: "122a5c3c-d745-469e-8478-d83d6ab6f86a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:22.624477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.624463 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:22.625973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.625948 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm" pod "122a5c3c-d745-469e-8478-d83d6ab6f86a" (UID: "122a5c3c-d745-469e-8478-d83d6ab6f86a"). InnerVolumeSpecName "kube-api-access-bzkrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:22.626085 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.626060 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca" pod "122a5c3c-d745-469e-8478-d83d6ab6f86a" (UID: "122a5c3c-d745-469e-8478-d83d6ab6f86a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:43:22.726067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.726040 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122a5c3c-d745-469e-8478-d83d6ab6f86a-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:22.726067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:22.726066 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzkrm\" (UniqueName: \"kubernetes.io/projected/122a5c3c-d745-469e-8478-d83d6ab6f86a-kube-api-access-bzkrm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:23.306609 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:23.306577 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" Nov 02 15:43:23.306793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:23.306577 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt" event={"ID":"122a5c3c-d745-469e-8478-d83d6ab6f86a","Type":"ContainerDied","Data":"f82f8675e72286a9c87528696c376750828f875151d0116a20c314be5c79524d"} Nov 02 15:43:23.306793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:23.306706 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f82f8675e72286a9c87528696c376750828f875151d0116a20c314be5c79524d" Nov 02 15:43:27.935143 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:43:27.935090 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:43:29.923094 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:29.923062 5091 scope.go:122] "RemoveContainer" containerID="24675509e1760a3b06621e81d0581b815a32f0838eb8ef3620b65bfc4c10c6af" Nov 02 15:43:29.932959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:29.932938 5091 scope.go:122] "RemoveContainer" containerID="2230bcd6d1a54be791ea1401a71c13e8b093c508c0a1e7770b6c5f92a5d036b8" Nov 02 15:43:29.943492 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:29.943465 5091 scope.go:122] "RemoveContainer" containerID="5497c8caa21f8664fb0d517442ddb962a2b9bebb2b4f008a55a6df1147e612af" Nov 02 15:43:29.952315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:29.952295 5091 scope.go:122] "RemoveContainer" containerID="d953057faebe197df3c9ac7c467b5b9f78b5e25f3cd64aab1a91e41afda37cc3" Nov 02 15:43:33.063533 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:33.063491 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f"] Nov 02 15:43:33.109623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:33.109588 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-891723-jm67f"] Nov 02 15:43:33.970733 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:33.970691 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaec6292-1884-43eb-a8cc-8c70fe9e84bb" path="/var/lib/kubelet/pods/aaec6292-1884-43eb-a8cc-8c70fe9e84bb/volumes" Nov 02 15:43:36.251128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:36.251094 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb"] Nov 02 15:43:36.251655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:36.251300 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" containerID="cri-o://d3e47317ffe9d95877fbe1c2498bf0a651841f9ba5585b3c3fcb9efba8efd3c3" gracePeriod=30 Nov 02 15:43:37.922947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:37.922911 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp"] Nov 02 15:43:37.923608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:37.923591 5091 cpu_manager.go:414] "containerMap: removing container" podUID="122a5c3c-d745-469e-8478-d83d6ab6f86a" containerName="revoke-keys" Nov 02 15:43:37.923668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:37.923610 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122a5c3c-d745-469e-8478-d83d6ab6f86a" containerName="revoke-keys" Nov 02 15:43:37.924602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:37.924572 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="122a5c3c-d745-469e-8478-d83d6ab6f86a" containerName="revoke-keys" Nov 02 15:43:37.931805 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:37.931782 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:37.982787 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:37.982753 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp"] Nov 02 15:43:38.022927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.022888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.023120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.023052 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85hb\" (UniqueName: \"kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.023203 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.023143 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.024724 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.024703 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:43:38.024802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.024789 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:43:38.124904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.124874 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.125060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.124947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85hb\" (UniqueName: \"kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.125060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.124973 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.125793 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.125768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.127310 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.127287 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.188422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.188349 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85hb\" (UniqueName: \"kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb\") pod \"maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.246567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.246533 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:38.642669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:38.642643 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp"] Nov 02 15:43:38.645965 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:43:38.645925 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b8e25b_69a6_43a7_a3c2_eff38f71d184.slice/crio-0bc108475f3bf36dea16d97a12b22993256804919a7694f26843082870beb677 WatchSource:0}: Error finding container 0bc108475f3bf36dea16d97a12b22993256804919a7694f26843082870beb677: Status 404 returned error can't find the container with id 0bc108475f3bf36dea16d97a12b22993256804919a7694f26843082870beb677 Nov 02 15:43:39.440607 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:39.440565 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" event={"ID":"22b8e25b-69a6-43a7-a3c2-eff38f71d184","Type":"ContainerStarted","Data":"2712957ecd1e0a64d3c74706efa8a6e3823f27243d6e196a50540a4184e72333"} Nov 02 15:43:39.441011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:39.440635 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" event={"ID":"22b8e25b-69a6-43a7-a3c2-eff38f71d184","Type":"ContainerStarted","Data":"0bc108475f3bf36dea16d97a12b22993256804919a7694f26843082870beb677"} Nov 02 15:43:39.556571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:39.556500 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" podStartSLOduration=2.5564854329999998 podStartE2EDuration="2.556485433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:43:39.554181787 +0000 UTC m=+2773.718734326" watchObservedRunningTime="2026-11-02 15:43:39.556485433 +0000 UTC m=+2773.721037972" Nov 02 15:43:40.449154 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:40.449117 5091 generic.go:356] "Generic (PLEG): container finished" podID="22b8e25b-69a6-43a7-a3c2-eff38f71d184" containerID="2712957ecd1e0a64d3c74706efa8a6e3823f27243d6e196a50540a4184e72333" exitCode=0 Nov 02 15:43:40.449620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:40.449202 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" event={"ID":"22b8e25b-69a6-43a7-a3c2-eff38f71d184","Type":"ContainerDied","Data":"2712957ecd1e0a64d3c74706efa8a6e3823f27243d6e196a50540a4184e72333"} Nov 02 15:43:40.676175 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:40.676143 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b"] Nov 02 15:43:40.676600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:40.676566 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" containerID="cri-o://b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a" gracePeriod=30 Nov 02 15:43:41.462666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.462083 5091 generic.go:356] "Generic (PLEG): container finished" podID="c090c586-ac91-41db-834c-629b79432c1f" containerID="d3e47317ffe9d95877fbe1c2498bf0a651841f9ba5585b3c3fcb9efba8efd3c3" exitCode=0 Nov 02 15:43:41.462666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.462155 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" event={"ID":"c090c586-ac91-41db-834c-629b79432c1f","Type":"ContainerDied","Data":"d3e47317ffe9d95877fbe1c2498bf0a651841f9ba5585b3c3fcb9efba8efd3c3"} Nov 02 15:43:41.539292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.539265 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:43:41.597880 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.597855 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.597900 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.597927 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.597962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.597996 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598149 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598177 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:41.598197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598184 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:41.598360 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598305 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598379 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert\") pod \"c090c586-ac91-41db-834c-629b79432c1f\" (UID: \"c090c586-ac91-41db-834c-629b79432c1f\") " Nov 02 15:43:41.598822 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598804 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.598860 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.598824 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.599452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.599087 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:41.599452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.599335 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:41.599452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.599408 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:41.601204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.601182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:43:41.601204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.601194 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "kube-api-access-rxdvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:41.601415 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.601398 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:41.601578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.601548 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token" pod "c090c586-ac91-41db-834c-629b79432c1f" (UID: "c090c586-ac91-41db-834c-629b79432c1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:41.631627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.631606 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:41.699679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.699641 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb\" (UniqueName: \"kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb\") pod \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " Nov 02 15:43:41.699850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.699752 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca\") pod \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " Nov 02 15:43:41.699850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.699821 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca\") pod \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\" (UID: \"22b8e25b-69a6-43a7-a3c2-eff38f71d184\") " Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700240 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700261 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700275 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c090c586-ac91-41db-834c-629b79432c1f-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700289 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdvc\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-kube-api-access-rxdvc\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700301 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca" pod "22b8e25b-69a6-43a7-a3c2-eff38f71d184" (UID: "22b8e25b-69a6-43a7-a3c2-eff38f71d184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700310 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c090c586-ac91-41db-834c-629b79432c1f-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700357 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c090c586-ac91-41db-834c-629b79432c1f-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.700396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.700375 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c090c586-ac91-41db-834c-629b79432c1f-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.703287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.702612 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca" pod "22b8e25b-69a6-43a7-a3c2-eff38f71d184" (UID: "22b8e25b-69a6-43a7-a3c2-eff38f71d184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:43:41.703287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.703120 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb" pod "22b8e25b-69a6-43a7-a3c2-eff38f71d184" (UID: "22b8e25b-69a6-43a7-a3c2-eff38f71d184"). InnerVolumeSpecName "kube-api-access-c85hb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:41.801363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.801280 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c85hb\" (UniqueName: \"kubernetes.io/projected/22b8e25b-69a6-43a7-a3c2-eff38f71d184-kube-api-access-c85hb\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.801363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.801320 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:41.801363 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:41.801338 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22b8e25b-69a6-43a7-a3c2-eff38f71d184-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:42.470842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.470816 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" Nov 02 15:43:42.470842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.470822 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp" event={"ID":"22b8e25b-69a6-43a7-a3c2-eff38f71d184","Type":"ContainerDied","Data":"0bc108475f3bf36dea16d97a12b22993256804919a7694f26843082870beb677"} Nov 02 15:43:42.471321 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.470873 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bc108475f3bf36dea16d97a12b22993256804919a7694f26843082870beb677" Nov 02 15:43:42.472569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.472542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" event={"ID":"c090c586-ac91-41db-834c-629b79432c1f","Type":"ContainerDied","Data":"fe004718aa0ffbfedacafdf47eff418f4293e106d5885b17c12f3dccdf20c6ed"} Nov 02 15:43:42.472569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.472560 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb" Nov 02 15:43:42.472747 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.472594 5091 scope.go:122] "RemoveContainer" containerID="d3e47317ffe9d95877fbe1c2498bf0a651841f9ba5585b3c3fcb9efba8efd3c3" Nov 02 15:43:42.682889 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.682830 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb"] Nov 02 15:43:42.686662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:42.686634 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tlctb"] Nov 02 15:43:43.960988 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:43.960955 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c090c586-ac91-41db-834c-629b79432c1f" path="/var/lib/kubelet/pods/c090c586-ac91-41db-834c-629b79432c1f/volumes" Nov 02 15:43:45.956871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:45.956803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:43:46.052472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052438 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052483 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052545 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052578 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052642 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052686 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052770 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052833 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.052927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052837 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:46.052927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052864 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:46.052927 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.052871 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data\") pod \"c5d472de-908f-4f2f-8440-411c4b84f2c4\" (UID: \"c5d472de-908f-4f2f-8440-411c4b84f2c4\") " Nov 02 15:43:46.053255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:46.053255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053195 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:43:46.053397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053378 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.053465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053403 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.053465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053419 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5d472de-908f-4f2f-8440-411c4b84f2c4-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.053465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053433 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.053682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.053658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:46.054851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.054820 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:43:46.055049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.055018 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:46.055205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.055185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "kube-api-access-smm28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:43:46.055275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.055233 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy" pod "c5d472de-908f-4f2f-8440-411c4b84f2c4" (UID: "c5d472de-908f-4f2f-8440-411c4b84f2c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:43:46.154715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.154677 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.154715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.154716 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.154911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.154731 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smm28\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-kube-api-access-smm28\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.154911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.154747 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5d472de-908f-4f2f-8440-411c4b84f2c4-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.154911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.154762 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5d472de-908f-4f2f-8440-411c4b84f2c4-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:43:46.502613 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.502579 5091 generic.go:356] "Generic (PLEG): container finished" podID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerID="b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a" exitCode=0 Nov 02 15:43:46.502788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.502636 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" Nov 02 15:43:46.502788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.502711 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" event={"ID":"c5d472de-908f-4f2f-8440-411c4b84f2c4","Type":"ContainerDied","Data":"b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a"} Nov 02 15:43:46.502788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.502746 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b" event={"ID":"c5d472de-908f-4f2f-8440-411c4b84f2c4","Type":"ContainerDied","Data":"cddd82dba37ea70fcc456eb9fd4385479d893e6d8c3ea8ca6353daa7c819b6e3"} Nov 02 15:43:46.502788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.502766 5091 scope.go:122] "RemoveContainer" containerID="b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a" Nov 02 15:43:46.514044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.514023 5091 scope.go:122] "RemoveContainer" containerID="b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a" Nov 02 15:43:46.514304 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:43:46.514288 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a\": container with ID starting with b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a not found: ID does not exist" containerID="b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a" Nov 02 15:43:46.514367 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.514312 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a"} err="failed to get container status \"b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a\": rpc error: code = NotFound desc = could not find container \"b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a\": container with ID starting with b40b1a5445cffab13bbc897333d3c68878bf151cfa54a2b55e7176eeeafb692a not found: ID does not exist" Nov 02 15:43:46.671993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.671957 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b"] Nov 02 15:43:46.677237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:46.677212 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-891723-openshift-default-65c9889b4c-lz92b"] Nov 02 15:43:47.960758 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:43:47.960721 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" path="/var/lib/kubelet/pods/c5d472de-908f-4f2f-8440-411c4b84f2c4/volumes" Nov 02 15:43:57.936072 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:43:57.936032 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:44:07.067815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:07.067778 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx"] Nov 02 15:44:07.109069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:07.109026 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b213b5a-kdmxx"] Nov 02 15:44:07.961181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:07.961143 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c006b75-3951-4ccb-afe4-e685f98dda8e" path="/var/lib/kubelet/pods/4c006b75-3951-4ccb-afe4-e685f98dda8e/volumes" Nov 02 15:44:08.065269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:08.065234 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt"] Nov 02 15:44:08.111309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:08.111273 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b18037ab-n5swt"] Nov 02 15:44:09.959967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:09.959933 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43587417-a6b1-4c85-8bed-0322bd778470" path="/var/lib/kubelet/pods/43587417-a6b1-4c85-8bed-0322bd778470/volumes" Nov 02 15:44:27.936159 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:44:27.936114 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:44:44.413105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:44.413019 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp"] Nov 02 15:44:44.418645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:44.418624 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2c35ee-98fxp"] Nov 02 15:44:45.962641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:45.962604 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab79d46a-301c-43c8-ba7e-c1cfb09d820b" path="/var/lib/kubelet/pods/ab79d46a-301c-43c8-ba7e-c1cfb09d820b/volumes" Nov 02 15:44:56.073580 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:56.073537 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f"] Nov 02 15:44:56.117923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:56.117890 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de2f5f75-btw2f"] Nov 02 15:44:56.120674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:56.120663 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg"] Nov 02 15:44:56.123141 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:56.123115 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6bc4f6-j22sg"] Nov 02 15:44:57.934999 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:44:57.934957 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:44:57.960789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:57.960754 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be724fa-40b0-4d27-8287-48106f479d25" path="/var/lib/kubelet/pods/8be724fa-40b0-4d27-8287-48106f479d25/volumes" Nov 02 15:44:57.961237 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:44:57.961220 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b301ba99-9256-44b7-8ccb-61aa17e0549d" path="/var/lib/kubelet/pods/b301ba99-9256-44b7-8ccb-61aa17e0549d/volumes" Nov 02 15:45:00.222220 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222175 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j"] Nov 02 15:45:00.222850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222831 5091 cpu_manager.go:414] "containerMap: removing container" podUID="22b8e25b-69a6-43a7-a3c2-eff38f71d184" containerName="revoke-keys" Nov 02 15:45:00.222850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222849 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b8e25b-69a6-43a7-a3c2-eff38f71d184" containerName="revoke-keys" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222861 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222866 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222883 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222888 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222976 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b8e25b-69a6-43a7-a3c2-eff38f71d184" containerName="revoke-keys" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222983 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d472de-908f-4f2f-8440-411c4b84f2c4" containerName="istio-proxy" Nov 02 15:45:00.222998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.222990 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c090c586-ac91-41db-834c-629b79432c1f" containerName="istio-proxy" Nov 02 15:45:00.228507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.228477 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw"] Nov 02 15:45:00.229630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.229602 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.233188 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.233167 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2"] Nov 02 15:45:00.233313 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.233269 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.237136 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.237116 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.250969 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.250944 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p"] Nov 02 15:45:00.255996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.255967 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.269311 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.269286 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc"] Nov 02 15:45:00.274156 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.273854 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.297169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.297150 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:45:00.310980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.310961 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g"] Nov 02 15:45:00.314928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.314910 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh"] Nov 02 15:45:00.315034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.315016 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.318770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.318748 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.329681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329640 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.329815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329678 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thkl7\" (UniqueName: \"kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.329815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329720 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.329998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329822 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.329998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329867 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7plm\" (UniqueName: \"kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm\") pod \"maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.329998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329911 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ntvk\" (UniqueName: \"kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.329998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329948 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drzmf\" (UniqueName: \"kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf\") pod \"maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.329998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.329982 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.330283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.330008 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.330283 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.330080 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwc42\" (UniqueName: \"kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42\") pod \"maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.335853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.335831 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j"] Nov 02 15:45:00.351606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.351579 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw"] Nov 02 15:45:00.354478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.354452 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p"] Nov 02 15:45:00.361289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.361260 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2"] Nov 02 15:45:00.361407 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.361318 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc"] Nov 02 15:45:00.411015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.410966 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g"] Nov 02 15:45:00.411204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.411036 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh"] Nov 02 15:45:00.430980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.430947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwc42\" (UniqueName: \"kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42\") pod \"maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.431120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.430997 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb6mf\" (UniqueName: \"kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf\") pod \"maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.431186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431151 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.431186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431174 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thkl7\" (UniqueName: \"kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.431299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.431299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431234 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsqnb\" (UniqueName: \"kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb\") pod \"maas-api-key-cleanup-29893905-mb9qh\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.431299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431287 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.431456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431332 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.431456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431393 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-mb9qh\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.431456 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431435 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7plm\" (UniqueName: \"kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm\") pod \"maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.431668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431483 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ntvk\" (UniqueName: \"kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.431668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431554 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drzmf\" (UniqueName: \"kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf\") pod \"maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.431668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.431668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.431616 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.432047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.432023 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.432114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.432096 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.432170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.432148 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.432318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.432301 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.432555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.432507 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.491161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.491088 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwc42\" (UniqueName: \"kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42\") pod \"maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.491586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.491559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7plm\" (UniqueName: \"kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm\") pod \"maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.493029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.492995 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thkl7\" (UniqueName: \"kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7\") pod \"maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.493204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.493185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drzmf\" (UniqueName: \"kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf\") pod \"maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.504387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.504363 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ntvk\" (UniqueName: \"kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk\") pod \"maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.533277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.533250 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb6mf\" (UniqueName: \"kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf\") pod \"maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.533373 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.533347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsqnb\" (UniqueName: \"kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb\") pod \"maas-api-key-cleanup-29893905-mb9qh\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.533421 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.533382 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.533482 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.533421 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-mb9qh\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.534033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.534010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.534102 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.534025 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca\") pod \"maas-api-key-cleanup-29893905-mb9qh\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.549709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.549689 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:00.561006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.560982 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:00.570253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.570233 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:00.581802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.581765 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:00.591007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.590972 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:00.594149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.593738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsqnb\" (UniqueName: \"kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb\") pod \"maas-api-key-cleanup-29893905-mb9qh\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.602575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.602501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb6mf\" (UniqueName: \"kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf\") pod \"maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.651879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.651652 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:00.670850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.670804 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:00.943075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:00.943046 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j"] Nov 02 15:45:00.945471 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:00.945439 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cc25ee_c935_4bbd_8424_11c2c0c25221.slice/crio-ce364dea838b128ce3ba4283cbf23d35c8bd55efe7d6c3ce5829b5ef7ff51a7a WatchSource:0}: Error finding container ce364dea838b128ce3ba4283cbf23d35c8bd55efe7d6c3ce5829b5ef7ff51a7a: Status 404 returned error can't find the container with id ce364dea838b128ce3ba4283cbf23d35c8bd55efe7d6c3ce5829b5ef7ff51a7a Nov 02 15:45:01.022836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.022794 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw"] Nov 02 15:45:01.025562 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:01.025534 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a9157f_e263_4a9e_b33f_fa004ab12832.slice/crio-ed478ae0b95059e2625295199381c9a86164b45c472a5136c7c20610ed956783 WatchSource:0}: Error finding container ed478ae0b95059e2625295199381c9a86164b45c472a5136c7c20610ed956783: Status 404 returned error can't find the container with id ed478ae0b95059e2625295199381c9a86164b45c472a5136c7c20610ed956783 Nov 02 15:45:01.046059 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.045989 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p"] Nov 02 15:45:01.048461 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:01.048432 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8f17b6_0b14_475d_b509_cb8121afaef9.slice/crio-5ffc3beefba08a49195d260c328a0a637a2dd07482654c7bf1429211dfd77420 WatchSource:0}: Error finding container 5ffc3beefba08a49195d260c328a0a637a2dd07482654c7bf1429211dfd77420: Status 404 returned error can't find the container with id 5ffc3beefba08a49195d260c328a0a637a2dd07482654c7bf1429211dfd77420 Nov 02 15:45:01.053735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.053711 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerStarted","Data":"ed478ae0b95059e2625295199381c9a86164b45c472a5136c7c20610ed956783"} Nov 02 15:45:01.055477 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.055450 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" event={"ID":"33cc25ee-c935-4bbd-8424-11c2c0c25221","Type":"ContainerStarted","Data":"ce364dea838b128ce3ba4283cbf23d35c8bd55efe7d6c3ce5829b5ef7ff51a7a"} Nov 02 15:45:01.088264 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.088238 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh"] Nov 02 15:45:01.092887 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:01.092861 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf821ee_c867_4361_8cd8_649ae39f9cce.slice/crio-50e6dacd0ad90f117856220cae2fb42b3fe6e7988a2853209b4b0d856bd6452a WatchSource:0}: Error finding container 50e6dacd0ad90f117856220cae2fb42b3fe6e7988a2853209b4b0d856bd6452a: Status 404 returned error can't find the container with id 50e6dacd0ad90f117856220cae2fb42b3fe6e7988a2853209b4b0d856bd6452a Nov 02 15:45:01.097292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.097271 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc"] Nov 02 15:45:01.115715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.115689 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g"] Nov 02 15:45:01.184577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:01.184549 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2"] Nov 02 15:45:01.231888 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:01.231792 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f5bdb7_eb68_4bcf_8d18_e60d3d0a1420.slice/crio-a2a3fe0142abc379b414020765e1257b3c9d29d9d6d0c9b0cee995f65c84ccb6 WatchSource:0}: Error finding container a2a3fe0142abc379b414020765e1257b3c9d29d9d6d0c9b0cee995f65c84ccb6: Status 404 returned error can't find the container with id a2a3fe0142abc379b414020765e1257b3c9d29d9d6d0c9b0cee995f65c84ccb6 Nov 02 15:45:02.064119 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.064088 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerStarted","Data":"c7c7d8fc477d3b1ab96210fb5b7aac37834654a36cdc9febe152138cdc1d139c"} Nov 02 15:45:02.065876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.065848 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerID="7da61aac9cbbd98e112281029e22f041931d74722b5ab8e75655e1af9dea40f7" exitCode=60 Nov 02 15:45:02.065993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.065943 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" event={"ID":"2bf821ee-c867-4361-8cd8-649ae39f9cce","Type":"ContainerDied","Data":"7da61aac9cbbd98e112281029e22f041931d74722b5ab8e75655e1af9dea40f7"} Nov 02 15:45:02.066062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.065998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" event={"ID":"2bf821ee-c867-4361-8cd8-649ae39f9cce","Type":"ContainerStarted","Data":"50e6dacd0ad90f117856220cae2fb42b3fe6e7988a2853209b4b0d856bd6452a"} Nov 02 15:45:02.066163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.066149 5091 scope.go:122] "RemoveContainer" containerID="7da61aac9cbbd98e112281029e22f041931d74722b5ab8e75655e1af9dea40f7" Nov 02 15:45:02.067871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.067848 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerStarted","Data":"33a7004985ff8f2953f9fca6456d51398a808800532c3b33a7691747f0ba0e40"} Nov 02 15:45:02.067951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.067896 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerStarted","Data":"c772af2382c44ba6b4f0b2688c4870f7cd51b946c538aa6f3f4b9e7bc84fcb09"} Nov 02 15:45:02.069593 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.069574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerStarted","Data":"abfa1deeffc4e57ad84b6c58b3b591282270ef3abd324b40b4687a00bc2d3ace"} Nov 02 15:45:02.069667 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.069618 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerStarted","Data":"8394639d3307a1c9535c4ca4dd07ead8a85db513d3cf9ffff68c3cfd7a269dd5"} Nov 02 15:45:02.071499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.071475 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerStarted","Data":"692534a695bb06be76b03df719224c7b995c0c1c8f33a31a925920fc17fad290"} Nov 02 15:45:02.071632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.071555 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerStarted","Data":"5ffc3beefba08a49195d260c328a0a637a2dd07482654c7bf1429211dfd77420"} Nov 02 15:45:02.076151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.073897 5091 generic.go:356] "Generic (PLEG): container finished" podID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerID="06d235fc4c2a3b89086e4854d0970135a67370419174a760d4f708e6599abaea" exitCode=60 Nov 02 15:45:02.076151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.074030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" event={"ID":"33cc25ee-c935-4bbd-8424-11c2c0c25221","Type":"ContainerDied","Data":"06d235fc4c2a3b89086e4854d0970135a67370419174a760d4f708e6599abaea"} Nov 02 15:45:02.076151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.074452 5091 scope.go:122] "RemoveContainer" containerID="06d235fc4c2a3b89086e4854d0970135a67370419174a760d4f708e6599abaea" Nov 02 15:45:02.077861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.077316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" event={"ID":"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420","Type":"ContainerStarted","Data":"c1b420254d00140672fde989089985edfcf176822ecb575731f5eece228e3764"} Nov 02 15:45:02.077861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.077362 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" event={"ID":"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420","Type":"ContainerStarted","Data":"a2a3fe0142abc379b414020765e1257b3c9d29d9d6d0c9b0cee995f65c84ccb6"} Nov 02 15:45:02.180628 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.180579 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" podStartSLOduration=2.180566326 podStartE2EDuration="2.180566326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.179128383 +0000 UTC m=+2856.343680914" watchObservedRunningTime="2026-11-02 15:45:02.180566326 +0000 UTC m=+2856.345118866" Nov 02 15:45:02.412135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.412083 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" podStartSLOduration=2.412067678 podStartE2EDuration="2.412067678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.409422738 +0000 UTC m=+2856.573975279" watchObservedRunningTime="2026-11-02 15:45:02.412067678 +0000 UTC m=+2856.576620217" Nov 02 15:45:02.530823 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.530774 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" podStartSLOduration=2.530760117 podStartE2EDuration="2.530760117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.528327996 +0000 UTC m=+2856.692880536" watchObservedRunningTime="2026-11-02 15:45:02.530760117 +0000 UTC m=+2856.695312661" Nov 02 15:45:02.643072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.643019 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" podStartSLOduration=2.643003804 podStartE2EDuration="2.643003804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.642346262 +0000 UTC m=+2856.806898802" watchObservedRunningTime="2026-11-02 15:45:02.643003804 +0000 UTC m=+2856.807556342" Nov 02 15:45:02.755061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:02.755013 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" podStartSLOduration=2.754998294 podStartE2EDuration="2.754998294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:02.753404211 +0000 UTC m=+2856.917956752" watchObservedRunningTime="2026-11-02 15:45:02.754998294 +0000 UTC m=+2856.919550835" Nov 02 15:45:03.087361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.087270 5091 generic.go:356] "Generic (PLEG): container finished" podID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerID="692534a695bb06be76b03df719224c7b995c0c1c8f33a31a925920fc17fad290" exitCode=60 Nov 02 15:45:03.087624 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.087583 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerDied","Data":"692534a695bb06be76b03df719224c7b995c0c1c8f33a31a925920fc17fad290"} Nov 02 15:45:03.087774 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.087762 5091 scope.go:122] "RemoveContainer" containerID="692534a695bb06be76b03df719224c7b995c0c1c8f33a31a925920fc17fad290" Nov 02 15:45:03.089651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.089625 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" event={"ID":"33cc25ee-c935-4bbd-8424-11c2c0c25221","Type":"ContainerStarted","Data":"6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8"} Nov 02 15:45:03.091406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.091381 5091 generic.go:356] "Generic (PLEG): container finished" podID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerID="c1b420254d00140672fde989089985edfcf176822ecb575731f5eece228e3764" exitCode=60 Nov 02 15:45:03.091499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.091463 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" event={"ID":"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420","Type":"ContainerDied","Data":"c1b420254d00140672fde989089985edfcf176822ecb575731f5eece228e3764"} Nov 02 15:45:03.091697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.091681 5091 scope.go:122] "RemoveContainer" containerID="c1b420254d00140672fde989089985edfcf176822ecb575731f5eece228e3764" Nov 02 15:45:03.093152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.093130 5091 generic.go:356] "Generic (PLEG): container finished" podID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerID="c7c7d8fc477d3b1ab96210fb5b7aac37834654a36cdc9febe152138cdc1d139c" exitCode=60 Nov 02 15:45:03.093245 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.093200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerDied","Data":"c7c7d8fc477d3b1ab96210fb5b7aac37834654a36cdc9febe152138cdc1d139c"} Nov 02 15:45:03.093388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.093372 5091 scope.go:122] "RemoveContainer" containerID="c7c7d8fc477d3b1ab96210fb5b7aac37834654a36cdc9febe152138cdc1d139c" Nov 02 15:45:03.095194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.095174 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" event={"ID":"2bf821ee-c867-4361-8cd8-649ae39f9cce","Type":"ContainerStarted","Data":"3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8"} Nov 02 15:45:03.096764 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.096744 5091 generic.go:356] "Generic (PLEG): container finished" podID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerID="33a7004985ff8f2953f9fca6456d51398a808800532c3b33a7691747f0ba0e40" exitCode=60 Nov 02 15:45:03.096853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.096802 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerDied","Data":"33a7004985ff8f2953f9fca6456d51398a808800532c3b33a7691747f0ba0e40"} Nov 02 15:45:03.097013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.097000 5091 scope.go:122] "RemoveContainer" containerID="33a7004985ff8f2953f9fca6456d51398a808800532c3b33a7691747f0ba0e40" Nov 02 15:45:03.098490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.098467 5091 generic.go:356] "Generic (PLEG): container finished" podID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerID="abfa1deeffc4e57ad84b6c58b3b591282270ef3abd324b40b4687a00bc2d3ace" exitCode=60 Nov 02 15:45:03.098612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.098546 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerDied","Data":"abfa1deeffc4e57ad84b6c58b3b591282270ef3abd324b40b4687a00bc2d3ace"} Nov 02 15:45:03.098751 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.098732 5091 scope.go:122] "RemoveContainer" containerID="abfa1deeffc4e57ad84b6c58b3b591282270ef3abd324b40b4687a00bc2d3ace" Nov 02 15:45:03.325231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.325182 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" podStartSLOduration=3.325168241 podStartE2EDuration="3.325168241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:03.321158367 +0000 UTC m=+2857.485710908" watchObservedRunningTime="2026-11-02 15:45:03.325168241 +0000 UTC m=+2857.489720780" Nov 02 15:45:03.560405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:03.560358 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" podStartSLOduration=3.560343757 podStartE2EDuration="3.560343757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:03.445949777 +0000 UTC m=+2857.610502318" watchObservedRunningTime="2026-11-02 15:45:03.560343757 +0000 UTC m=+2857.724896299" Nov 02 15:45:04.106997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.106971 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerStarted","Data":"8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798"} Nov 02 15:45:04.108856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.108833 5091 generic.go:356] "Generic (PLEG): container finished" podID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerID="6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8" exitCode=60 Nov 02 15:45:04.108972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.108912 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" event={"ID":"33cc25ee-c935-4bbd-8424-11c2c0c25221","Type":"ContainerDied","Data":"6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8"} Nov 02 15:45:04.108972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.108963 5091 scope.go:122] "RemoveContainer" containerID="06d235fc4c2a3b89086e4854d0970135a67370419174a760d4f708e6599abaea" Nov 02 15:45:04.109172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.109152 5091 scope.go:122] "RemoveContainer" containerID="6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8" Nov 02 15:45:04.109455 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:04.109429 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j_odh-ai-gateway-infra(33cc25ee-c935-4bbd-8424-11c2c0c25221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" Nov 02 15:45:04.110958 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.110940 5091 generic.go:356] "Generic (PLEG): container finished" podID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerID="6ee3114e44dc6cbbd318508a9fbf6a99251667eb4b6cdc23cf23caf64202f4a9" exitCode=60 Nov 02 15:45:04.111095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.111023 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" event={"ID":"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420","Type":"ContainerDied","Data":"6ee3114e44dc6cbbd318508a9fbf6a99251667eb4b6cdc23cf23caf64202f4a9"} Nov 02 15:45:04.111263 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.111250 5091 scope.go:122] "RemoveContainer" containerID="6ee3114e44dc6cbbd318508a9fbf6a99251667eb4b6cdc23cf23caf64202f4a9" Nov 02 15:45:04.111565 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:04.111510 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2_odh-ai-gateway-infra(e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" Nov 02 15:45:04.113036 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.113016 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerStarted","Data":"077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075"} Nov 02 15:45:04.116295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.116259 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerID="3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8" exitCode=60 Nov 02 15:45:04.116419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.116402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" event={"ID":"2bf821ee-c867-4361-8cd8-649ae39f9cce","Type":"ContainerDied","Data":"3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8"} Nov 02 15:45:04.116783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.116765 5091 scope.go:122] "RemoveContainer" containerID="3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8" Nov 02 15:45:04.117031 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:04.117010 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-mb9qh_odh-ai-gateway-infra(2bf821ee-c867-4361-8cd8-649ae39f9cce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" Nov 02 15:45:04.118983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.118960 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerStarted","Data":"9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848"} Nov 02 15:45:04.120964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.120942 5091 generic.go:356] "Generic (PLEG): container finished" podID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerID="54c563efb8fb2dafabedf6c2427733e04d39d40d9f095cc01737aa1ee1dd139c" exitCode=60 Nov 02 15:45:04.121048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.121026 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerDied","Data":"54c563efb8fb2dafabedf6c2427733e04d39d40d9f095cc01737aa1ee1dd139c"} Nov 02 15:45:04.121189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.121176 5091 scope.go:122] "RemoveContainer" containerID="54c563efb8fb2dafabedf6c2427733e04d39d40d9f095cc01737aa1ee1dd139c" Nov 02 15:45:04.121352 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:04.121336 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc_odh-ai-gateway-infra(a6689e87-a087-4b3c-836a-a4cb386ae761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" Nov 02 15:45:04.137472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.137451 5091 scope.go:122] "RemoveContainer" containerID="c1b420254d00140672fde989089985edfcf176822ecb575731f5eece228e3764" Nov 02 15:45:04.150341 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.150322 5091 scope.go:122] "RemoveContainer" containerID="7da61aac9cbbd98e112281029e22f041931d74722b5ab8e75655e1af9dea40f7" Nov 02 15:45:04.162712 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.162696 5091 scope.go:122] "RemoveContainer" containerID="abfa1deeffc4e57ad84b6c58b3b591282270ef3abd324b40b4687a00bc2d3ace" Nov 02 15:45:04.752068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:04.752029 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8"] Nov 02 15:45:05.132543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.132495 5091 generic.go:356] "Generic (PLEG): container finished" podID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerID="9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848" exitCode=60 Nov 02 15:45:05.132748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.132585 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerDied","Data":"9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848"} Nov 02 15:45:05.132748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.132639 5091 scope.go:122] "RemoveContainer" containerID="33a7004985ff8f2953f9fca6456d51398a808800532c3b33a7691747f0ba0e40" Nov 02 15:45:05.133070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.133048 5091 scope.go:122] "RemoveContainer" containerID="9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848" Nov 02 15:45:05.133399 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.133360 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g_odh-ai-gateway-infra(c96a77e4-8d39-4ea9-9b7f-9967675d23ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" Nov 02 15:45:05.136296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.136272 5091 scope.go:122] "RemoveContainer" containerID="54c563efb8fb2dafabedf6c2427733e04d39d40d9f095cc01737aa1ee1dd139c" Nov 02 15:45:05.136559 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.136533 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc_odh-ai-gateway-infra(a6689e87-a087-4b3c-836a-a4cb386ae761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" Nov 02 15:45:05.138792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.138764 5091 generic.go:356] "Generic (PLEG): container finished" podID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerID="8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798" exitCode=60 Nov 02 15:45:05.138904 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.138840 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerDied","Data":"8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798"} Nov 02 15:45:05.139275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.139253 5091 scope.go:122] "RemoveContainer" containerID="8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798" Nov 02 15:45:05.139750 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.139550 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p_odh-ai-gateway-infra(bd8f17b6-0b14-475d-b509-cb8121afaef9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" Nov 02 15:45:05.142128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.142107 5091 scope.go:122] "RemoveContainer" containerID="6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8" Nov 02 15:45:05.142533 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.142493 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j_odh-ai-gateway-infra(33cc25ee-c935-4bbd-8424-11c2c0c25221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" Nov 02 15:45:05.146563 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.146538 5091 scope.go:122] "RemoveContainer" containerID="6ee3114e44dc6cbbd318508a9fbf6a99251667eb4b6cdc23cf23caf64202f4a9" Nov 02 15:45:05.146778 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.146755 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2_odh-ai-gateway-infra(e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" Nov 02 15:45:05.148584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.148559 5091 generic.go:356] "Generic (PLEG): container finished" podID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerID="077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075" exitCode=60 Nov 02 15:45:05.148826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.148796 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerDied","Data":"077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075"} Nov 02 15:45:05.148939 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.148909 5091 scope.go:122] "RemoveContainer" containerID="077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075" Nov 02 15:45:05.149156 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.149132 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw_odh-ai-gateway-infra(07a9157f-e263-4a9e-b33f-fa004ab12832)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" Nov 02 15:45:05.150586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.150563 5091 scope.go:122] "RemoveContainer" containerID="692534a695bb06be76b03df719224c7b995c0c1c8f33a31a925920fc17fad290" Nov 02 15:45:05.150767 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.150751 5091 scope.go:122] "RemoveContainer" containerID="3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8" Nov 02 15:45:05.150944 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:05.150925 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-mb9qh_odh-ai-gateway-infra(2bf821ee-c867-4361-8cd8-649ae39f9cce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" Nov 02 15:45:05.164230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:05.164204 5091 scope.go:122] "RemoveContainer" containerID="c7c7d8fc477d3b1ab96210fb5b7aac37834654a36cdc9febe152138cdc1d139c" Nov 02 15:45:06.160864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:06.160829 5091 scope.go:122] "RemoveContainer" containerID="8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798" Nov 02 15:45:06.161333 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:06.161189 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p_odh-ai-gateway-infra(bd8f17b6-0b14-475d-b509-cb8121afaef9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" Nov 02 15:45:06.162674 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:06.162646 5091 scope.go:122] "RemoveContainer" containerID="077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075" Nov 02 15:45:06.162978 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:06.162950 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw_odh-ai-gateway-infra(07a9157f-e263-4a9e-b33f-fa004ab12832)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" Nov 02 15:45:06.164550 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:06.164505 5091 scope.go:122] "RemoveContainer" containerID="9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848" Nov 02 15:45:06.164802 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:06.164783 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g_odh-ai-gateway-infra(c96a77e4-8d39-4ea9-9b7f-9967675d23ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" Nov 02 15:45:09.753390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:09.753315 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" podUID="5cffcab8-f4c0-4d7e-ab2f-06119f06149e" containerName="payload-pre-processing" containerID="cri-o://b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685" gracePeriod=25 Nov 02 15:45:10.050693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.050667 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:45:10.172016 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.171981 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4\" (UniqueName: \"kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4\") pod \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " Nov 02 15:45:10.172215 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.172108 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume\") pod \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\" (UID: \"5cffcab8-f4c0-4d7e-ab2f-06119f06149e\") " Nov 02 15:45:10.172554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.172507 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume" pod "5cffcab8-f4c0-4d7e-ab2f-06119f06149e" (UID: "5cffcab8-f4c0-4d7e-ab2f-06119f06149e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:10.174773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.174740 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4" pod "5cffcab8-f4c0-4d7e-ab2f-06119f06149e" (UID: "5cffcab8-f4c0-4d7e-ab2f-06119f06149e"). InnerVolumeSpecName "kube-api-access-f7xg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:10.198620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.198590 5091 generic.go:356] "Generic (PLEG): container finished" podID="5cffcab8-f4c0-4d7e-ab2f-06119f06149e" containerID="b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685" exitCode=0 Nov 02 15:45:10.198732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.198636 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" Nov 02 15:45:10.198732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.198646 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" event={"ID":"5cffcab8-f4c0-4d7e-ab2f-06119f06149e","Type":"ContainerDied","Data":"b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685"} Nov 02 15:45:10.198732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.198685 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8" event={"ID":"5cffcab8-f4c0-4d7e-ab2f-06119f06149e","Type":"ContainerDied","Data":"3d7425a8e5a091956e03b8809d14c3fe372f70fa14b6e383b9f34500424bc434"} Nov 02 15:45:10.198732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.198705 5091 scope.go:122] "RemoveContainer" containerID="b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685" Nov 02 15:45:10.210069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.210046 5091 scope.go:122] "RemoveContainer" containerID="b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685" Nov 02 15:45:10.210394 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:10.210374 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685\": container with ID starting with b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685 not found: ID does not exist" containerID="b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685" Nov 02 15:45:10.210463 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.210401 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685"} err="failed to get container status \"b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685\": rpc error: code = NotFound desc = could not find container \"b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685\": container with ID starting with b5ca44af2f7699606f73fd280a4836470f66f63db59a874a884ac4eaa2c5b685 not found: ID does not exist" Nov 02 15:45:10.273627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.273542 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7xg4\" (UniqueName: \"kubernetes.io/projected/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-kube-api-access-f7xg4\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:10.273627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.273573 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cffcab8-f4c0-4d7e-ab2f-06119f06149e-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:10.364972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.364935 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8"] Nov 02 15:45:10.370458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:10.370419 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3b351-56fb44c9b7-gw4b8"] Nov 02 15:45:11.961144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:11.961109 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cffcab8-f4c0-4d7e-ab2f-06119f06149e" path="/var/lib/kubelet/pods/5cffcab8-f4c0-4d7e-ab2f-06119f06149e/volumes" Nov 02 15:45:14.953219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:14.953183 5091 scope.go:122] "RemoveContainer" containerID="9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848" Nov 02 15:45:14.953785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:14.953240 5091 scope.go:122] "RemoveContainer" containerID="077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075" Nov 02 15:45:14.953785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:14.953401 5091 scope.go:122] "RemoveContainer" containerID="3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8" Nov 02 15:45:15.956750 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:15.956721 5091 scope.go:122] "RemoveContainer" containerID="6ee3114e44dc6cbbd318508a9fbf6a99251667eb4b6cdc23cf23caf64202f4a9" Nov 02 15:45:16.245039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.244760 5091 generic.go:356] "Generic (PLEG): container finished" podID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerID="f48f387ddd8ffef8ae2d12da5da13f535fceff2779dd00f16995bb31f95a5c2d" exitCode=60 Nov 02 15:45:16.245039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.244906 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerDied","Data":"f48f387ddd8ffef8ae2d12da5da13f535fceff2779dd00f16995bb31f95a5c2d"} Nov 02 15:45:16.245039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.244968 5091 scope.go:122] "RemoveContainer" containerID="077f8782922c16223390f1b4e3bbd1520cc57a0bf4bff92e77c85096572c2075" Nov 02 15:45:16.245375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.245353 5091 scope.go:122] "RemoveContainer" containerID="f48f387ddd8ffef8ae2d12da5da13f535fceff2779dd00f16995bb31f95a5c2d" Nov 02 15:45:16.245683 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:16.245651 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw_odh-ai-gateway-infra(07a9157f-e263-4a9e-b33f-fa004ab12832)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" Nov 02 15:45:16.254735 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.254709 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerID="c1175e88f967a066806067ca245cc56c81d5907119c09acd52a3dcd605e812d7" exitCode=60 Nov 02 15:45:16.254844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.254786 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" event={"ID":"2bf821ee-c867-4361-8cd8-649ae39f9cce","Type":"ContainerDied","Data":"c1175e88f967a066806067ca245cc56c81d5907119c09acd52a3dcd605e812d7"} Nov 02 15:45:16.255132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.255106 5091 scope.go:122] "RemoveContainer" containerID="c1175e88f967a066806067ca245cc56c81d5907119c09acd52a3dcd605e812d7" Nov 02 15:45:16.255360 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:16.255344 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29893905-mb9qh_odh-ai-gateway-infra(2bf821ee-c867-4361-8cd8-649ae39f9cce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" Nov 02 15:45:16.257103 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.257082 5091 generic.go:356] "Generic (PLEG): container finished" podID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerID="04f3148f0abe9788d9fccab976e12790f04994bbdfcbb2e31e55f1de51ad1805" exitCode=60 Nov 02 15:45:16.257190 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.257156 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerDied","Data":"04f3148f0abe9788d9fccab976e12790f04994bbdfcbb2e31e55f1de51ad1805"} Nov 02 15:45:16.257468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.257455 5091 scope.go:122] "RemoveContainer" containerID="04f3148f0abe9788d9fccab976e12790f04994bbdfcbb2e31e55f1de51ad1805" Nov 02 15:45:16.257728 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:16.257710 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g_odh-ai-gateway-infra(c96a77e4-8d39-4ea9-9b7f-9967675d23ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" Nov 02 15:45:16.265840 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.265816 5091 scope.go:122] "RemoveContainer" containerID="3b02babe7381b550200277eeea84d6fb22ac1896fb8be53a8535fd3c97ce03f8" Nov 02 15:45:16.279090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.279072 5091 scope.go:122] "RemoveContainer" containerID="9efa67f834ece9058ca7b031a2dc13d9ad88ce4a9ba5800f0a2bcab201e3c848" Nov 02 15:45:16.952941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:16.952908 5091 scope.go:122] "RemoveContainer" containerID="54c563efb8fb2dafabedf6c2427733e04d39d40d9f095cc01737aa1ee1dd139c" Nov 02 15:45:17.265813 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.265780 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerStarted","Data":"03e5f1338f779f51904c87025a37df7225ea24decdf70ce5d7818cedb823c795"} Nov 02 15:45:17.267841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.267820 5091 generic.go:356] "Generic (PLEG): container finished" podID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerID="5040756abc13d61dd05a0f285b91155eca50938ae8c2883bfb028c9f287b3687" exitCode=60 Nov 02 15:45:17.267949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.267886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" event={"ID":"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420","Type":"ContainerDied","Data":"5040756abc13d61dd05a0f285b91155eca50938ae8c2883bfb028c9f287b3687"} Nov 02 15:45:17.267949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.267932 5091 scope.go:122] "RemoveContainer" containerID="6ee3114e44dc6cbbd318508a9fbf6a99251667eb4b6cdc23cf23caf64202f4a9" Nov 02 15:45:17.268252 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.268236 5091 scope.go:122] "RemoveContainer" containerID="5040756abc13d61dd05a0f285b91155eca50938ae8c2883bfb028c9f287b3687" Nov 02 15:45:17.268505 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:17.268480 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2_odh-ai-gateway-infra(e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" Nov 02 15:45:17.370788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.370757 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw"] Nov 02 15:45:17.485400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.485321 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh"] Nov 02 15:45:17.532191 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.532164 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:17.558467 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.558441 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca\") pod \"07a9157f-e263-4a9e-b33f-fa004ab12832\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " Nov 02 15:45:17.558681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.558662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm\" (UniqueName: \"kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm\") pod \"07a9157f-e263-4a9e-b33f-fa004ab12832\" (UID: \"07a9157f-e263-4a9e-b33f-fa004ab12832\") " Nov 02 15:45:17.558924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.558897 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca" pod "07a9157f-e263-4a9e-b33f-fa004ab12832" (UID: "07a9157f-e263-4a9e-b33f-fa004ab12832"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:17.559074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.559052 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/07a9157f-e263-4a9e-b33f-fa004ab12832-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.561453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.561422 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm" pod "07a9157f-e263-4a9e-b33f-fa004ab12832" (UID: "07a9157f-e263-4a9e-b33f-fa004ab12832"). InnerVolumeSpecName "kube-api-access-n7plm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:17.606438 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.606408 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g"] Nov 02 15:45:17.638825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.638797 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:17.660101 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.660067 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca\") pod \"2bf821ee-c867-4361-8cd8-649ae39f9cce\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " Nov 02 15:45:17.660258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.660195 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb\" (UniqueName: \"kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb\") pod \"2bf821ee-c867-4361-8cd8-649ae39f9cce\" (UID: \"2bf821ee-c867-4361-8cd8-649ae39f9cce\") " Nov 02 15:45:17.660499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.660469 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca" pod "2bf821ee-c867-4361-8cd8-649ae39f9cce" (UID: "2bf821ee-c867-4361-8cd8-649ae39f9cce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:17.661274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.660736 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7plm\" (UniqueName: \"kubernetes.io/projected/07a9157f-e263-4a9e-b33f-fa004ab12832-kube-api-access-n7plm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.661274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.660758 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bf821ee-c867-4361-8cd8-649ae39f9cce-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.662464 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.662439 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb" pod "2bf821ee-c867-4361-8cd8-649ae39f9cce" (UID: "2bf821ee-c867-4361-8cd8-649ae39f9cce"). InnerVolumeSpecName "kube-api-access-hsqnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:17.754962 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.754898 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:17.761839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.761814 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca\") pod \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " Nov 02 15:45:17.761972 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.761884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf\" (UniqueName: \"kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf\") pod \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\" (UID: \"c96a77e4-8d39-4ea9-9b7f-9967675d23ff\") " Nov 02 15:45:17.762099 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.762087 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsqnb\" (UniqueName: \"kubernetes.io/projected/2bf821ee-c867-4361-8cd8-649ae39f9cce-kube-api-access-hsqnb\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.762170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.762123 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca" pod "c96a77e4-8d39-4ea9-9b7f-9967675d23ff" (UID: "c96a77e4-8d39-4ea9-9b7f-9967675d23ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:17.763886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.763861 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf" pod "c96a77e4-8d39-4ea9-9b7f-9967675d23ff" (UID: "c96a77e4-8d39-4ea9-9b7f-9967675d23ff"). InnerVolumeSpecName "kube-api-access-wb6mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:17.863211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.863183 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb6mf\" (UniqueName: \"kubernetes.io/projected/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-kube-api-access-wb6mf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.863211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.863210 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c96a77e4-8d39-4ea9-9b7f-9967675d23ff-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:17.952806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:17.952770 5091 scope.go:122] "RemoveContainer" containerID="6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8" Nov 02 15:45:18.283932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.283849 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" Nov 02 15:45:18.284377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.283851 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw" event={"ID":"07a9157f-e263-4a9e-b33f-fa004ab12832","Type":"ContainerDied","Data":"ed478ae0b95059e2625295199381c9a86164b45c472a5136c7c20610ed956783"} Nov 02 15:45:18.284377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.284016 5091 scope.go:122] "RemoveContainer" containerID="f48f387ddd8ffef8ae2d12da5da13f535fceff2779dd00f16995bb31f95a5c2d" Nov 02 15:45:18.285365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.285338 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" Nov 02 15:45:18.285510 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.285339 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh" event={"ID":"2bf821ee-c867-4361-8cd8-649ae39f9cce","Type":"ContainerDied","Data":"50e6dacd0ad90f117856220cae2fb42b3fe6e7988a2853209b4b0d856bd6452a"} Nov 02 15:45:18.287048 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.287030 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" Nov 02 15:45:18.287132 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.287050 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g" event={"ID":"c96a77e4-8d39-4ea9-9b7f-9967675d23ff","Type":"ContainerDied","Data":"c772af2382c44ba6b4f0b2688c4870f7cd51b946c538aa6f3f4b9e7bc84fcb09"} Nov 02 15:45:18.289445 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.289423 5091 generic.go:356] "Generic (PLEG): container finished" podID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerID="03e5f1338f779f51904c87025a37df7225ea24decdf70ce5d7818cedb823c795" exitCode=60 Nov 02 15:45:18.289568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.289453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerDied","Data":"03e5f1338f779f51904c87025a37df7225ea24decdf70ce5d7818cedb823c795"} Nov 02 15:45:18.289869 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.289853 5091 scope.go:122] "RemoveContainer" containerID="03e5f1338f779f51904c87025a37df7225ea24decdf70ce5d7818cedb823c795" Nov 02 15:45:18.290130 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:18.290111 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc_odh-ai-gateway-infra(a6689e87-a087-4b3c-836a-a4cb386ae761)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" Nov 02 15:45:18.298094 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.298070 5091 scope.go:122] "RemoveContainer" containerID="c1175e88f967a066806067ca245cc56c81d5907119c09acd52a3dcd605e812d7" Nov 02 15:45:18.307306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.307286 5091 scope.go:122] "RemoveContainer" containerID="04f3148f0abe9788d9fccab976e12790f04994bbdfcbb2e31e55f1de51ad1805" Nov 02 15:45:18.317249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.317228 5091 scope.go:122] "RemoveContainer" containerID="54c563efb8fb2dafabedf6c2427733e04d39d40d9f095cc01737aa1ee1dd139c" Nov 02 15:45:18.401107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.401076 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc"] Nov 02 15:45:18.444500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.444463 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw"] Nov 02 15:45:18.449587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.449559 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2203c8-29893905-fpsxw"] Nov 02 15:45:18.527032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.526996 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2"] Nov 02 15:45:18.557095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.556300 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g"] Nov 02 15:45:18.566174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.566138 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-9ca2cf-29893905-74n4g"] Nov 02 15:45:18.688423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.688398 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:18.776596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.776568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7\" (UniqueName: \"kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7\") pod \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " Nov 02 15:45:18.776737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.776620 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca\") pod \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\" (UID: \"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420\") " Nov 02 15:45:18.777038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.777010 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca" pod "e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" (UID: "e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:18.777159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.777138 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:18.778668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.778643 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7" pod "e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" (UID: "e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420"). InnerVolumeSpecName "kube-api-access-thkl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:18.785858 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.785827 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh"] Nov 02 15:45:18.791305 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.791283 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29893905-mb9qh"] Nov 02 15:45:18.878041 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.878015 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thkl7\" (UniqueName: \"kubernetes.io/projected/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420-kube-api-access-thkl7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:18.952983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:18.952960 5091 scope.go:122] "RemoveContainer" containerID="8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798" Nov 02 15:45:19.300084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.300049 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerStarted","Data":"40831df68390c031c1ce8147b266b6e9f71e3b55f455f4bd8582cb59fd4217b9"} Nov 02 15:45:19.302041 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.302018 5091 generic.go:356] "Generic (PLEG): container finished" podID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerID="3cdfeeb4e7bf7b03c75616635022c1154132e021cdafa6f4ed5e3f9c90fb9493" exitCode=60 Nov 02 15:45:19.302166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.302086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" event={"ID":"33cc25ee-c935-4bbd-8424-11c2c0c25221","Type":"ContainerDied","Data":"3cdfeeb4e7bf7b03c75616635022c1154132e021cdafa6f4ed5e3f9c90fb9493"} Nov 02 15:45:19.302166 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.302138 5091 scope.go:122] "RemoveContainer" containerID="6b621ee6e43ad066f7eea6f6700b8e65706d8207f33b0b59aa2008e219542bb8" Nov 02 15:45:19.302499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.302483 5091 scope.go:122] "RemoveContainer" containerID="3cdfeeb4e7bf7b03c75616635022c1154132e021cdafa6f4ed5e3f9c90fb9493" Nov 02 15:45:19.302815 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:19.302789 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j_odh-ai-gateway-infra(33cc25ee-c935-4bbd-8424-11c2c0c25221)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" Nov 02 15:45:19.303478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.303460 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" Nov 02 15:45:19.303620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.303484 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2" event={"ID":"e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420","Type":"ContainerDied","Data":"a2a3fe0142abc379b414020765e1257b3c9d29d9d6d0c9b0cee995f65c84ccb6"} Nov 02 15:45:19.315444 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.315428 5091 scope.go:122] "RemoveContainer" containerID="5040756abc13d61dd05a0f285b91155eca50938ae8c2883bfb028c9f287b3687" Nov 02 15:45:19.473345 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.473318 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:19.484991 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.484961 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca\") pod \"a6689e87-a087-4b3c-836a-a4cb386ae761\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " Nov 02 15:45:19.485149 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.485131 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42\" (UniqueName: \"kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42\") pod \"a6689e87-a087-4b3c-836a-a4cb386ae761\" (UID: \"a6689e87-a087-4b3c-836a-a4cb386ae761\") " Nov 02 15:45:19.485266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.485244 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca" pod "a6689e87-a087-4b3c-836a-a4cb386ae761" (UID: "a6689e87-a087-4b3c-836a-a4cb386ae761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:19.485560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.485506 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6689e87-a087-4b3c-836a-a4cb386ae761-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:19.487128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.487104 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42" pod "a6689e87-a087-4b3c-836a-a4cb386ae761" (UID: "a6689e87-a087-4b3c-836a-a4cb386ae761"). InnerVolumeSpecName "kube-api-access-kwc42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:19.529600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.529533 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2"] Nov 02 15:45:19.533725 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.533699 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-e98976-29893905-bkvd2"] Nov 02 15:45:19.586485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.586444 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwc42\" (UniqueName: \"kubernetes.io/projected/a6689e87-a087-4b3c-836a-a4cb386ae761-kube-api-access-kwc42\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:19.960626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.960594 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" path="/var/lib/kubelet/pods/07a9157f-e263-4a9e-b33f-fa004ab12832/volumes" Nov 02 15:45:19.961022 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.961009 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" path="/var/lib/kubelet/pods/2bf821ee-c867-4361-8cd8-649ae39f9cce/volumes" Nov 02 15:45:19.961376 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.961364 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" path="/var/lib/kubelet/pods/c96a77e4-8d39-4ea9-9b7f-9967675d23ff/volumes" Nov 02 15:45:19.962072 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:19.962056 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" path="/var/lib/kubelet/pods/e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420/volumes" Nov 02 15:45:20.315223 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.315192 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" Nov 02 15:45:20.315732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.315193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc" event={"ID":"a6689e87-a087-4b3c-836a-a4cb386ae761","Type":"ContainerDied","Data":"8394639d3307a1c9535c4ca4dd07ead8a85db513d3cf9ffff68c3cfd7a269dd5"} Nov 02 15:45:20.315732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.315370 5091 scope.go:122] "RemoveContainer" containerID="03e5f1338f779f51904c87025a37df7225ea24decdf70ce5d7818cedb823c795" Nov 02 15:45:20.317584 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.317552 5091 generic.go:356] "Generic (PLEG): container finished" podID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerID="40831df68390c031c1ce8147b266b6e9f71e3b55f455f4bd8582cb59fd4217b9" exitCode=60 Nov 02 15:45:20.317685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.317630 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerDied","Data":"40831df68390c031c1ce8147b266b6e9f71e3b55f455f4bd8582cb59fd4217b9"} Nov 02 15:45:20.318041 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.318016 5091 scope.go:122] "RemoveContainer" containerID="40831df68390c031c1ce8147b266b6e9f71e3b55f455f4bd8582cb59fd4217b9" Nov 02 15:45:20.318339 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:20.318312 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p_odh-ai-gateway-infra(bd8f17b6-0b14-475d-b509-cb8121afaef9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" Nov 02 15:45:20.325854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.325833 5091 scope.go:122] "RemoveContainer" containerID="8222c149edfebd1d866d49f83aec7e56e597ea1846a9c186e7768c8325ec0798" Nov 02 15:45:20.473648 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.473606 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc"] Nov 02 15:45:20.480997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.480966 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe8ef1-29893905-5qtrc"] Nov 02 15:45:20.661484 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.661455 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j"] Nov 02 15:45:20.777032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.777001 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p"] Nov 02 15:45:20.817424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.817401 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:20.901236 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.901206 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca\") pod \"33cc25ee-c935-4bbd-8424-11c2c0c25221\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " Nov 02 15:45:20.901398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.901354 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk\" (UniqueName: \"kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk\") pod \"33cc25ee-c935-4bbd-8424-11c2c0c25221\" (UID: \"33cc25ee-c935-4bbd-8424-11c2c0c25221\") " Nov 02 15:45:20.901573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.901551 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca" pod "33cc25ee-c935-4bbd-8424-11c2c0c25221" (UID: "33cc25ee-c935-4bbd-8424-11c2c0c25221"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:20.901682 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.901668 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33cc25ee-c935-4bbd-8424-11c2c0c25221-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:20.903396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:20.903374 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk" pod "33cc25ee-c935-4bbd-8424-11c2c0c25221" (UID: "33cc25ee-c935-4bbd-8424-11c2c0c25221"). InnerVolumeSpecName "kube-api-access-4ntvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.003014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.002983 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ntvk\" (UniqueName: \"kubernetes.io/projected/33cc25ee-c935-4bbd-8424-11c2c0c25221-kube-api-access-4ntvk\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.334081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.334046 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" event={"ID":"33cc25ee-c935-4bbd-8424-11c2c0c25221","Type":"ContainerDied","Data":"ce364dea838b128ce3ba4283cbf23d35c8bd55efe7d6c3ce5829b5ef7ff51a7a"} Nov 02 15:45:21.334478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.334103 5091 scope.go:122] "RemoveContainer" containerID="3cdfeeb4e7bf7b03c75616635022c1154132e021cdafa6f4ed5e3f9c90fb9493" Nov 02 15:45:21.334478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.334181 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j" Nov 02 15:45:21.545338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.545297 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:21.568025 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.567996 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j"] Nov 02 15:45:21.574342 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.574313 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fce776-29893905-fj77j"] Nov 02 15:45:21.611067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.610987 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca\") pod \"bd8f17b6-0b14-475d-b509-cb8121afaef9\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " Nov 02 15:45:21.611247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.611230 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf\" (UniqueName: \"kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf\") pod \"bd8f17b6-0b14-475d-b509-cb8121afaef9\" (UID: \"bd8f17b6-0b14-475d-b509-cb8121afaef9\") " Nov 02 15:45:21.611448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.611419 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca" pod "bd8f17b6-0b14-475d-b509-cb8121afaef9" (UID: "bd8f17b6-0b14-475d-b509-cb8121afaef9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:21.611689 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.611658 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd8f17b6-0b14-475d-b509-cb8121afaef9-maas-api-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.613409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.613381 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf" pod "bd8f17b6-0b14-475d-b509-cb8121afaef9" (UID: "bd8f17b6-0b14-475d-b509-cb8121afaef9"). InnerVolumeSpecName "kube-api-access-drzmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:21.712947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.712914 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drzmf\" (UniqueName: \"kubernetes.io/projected/bd8f17b6-0b14-475d-b509-cb8121afaef9-kube-api-access-drzmf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:21.963545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.963495 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" path="/var/lib/kubelet/pods/33cc25ee-c935-4bbd-8424-11c2c0c25221/volumes" Nov 02 15:45:21.964021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:21.964000 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" path="/var/lib/kubelet/pods/a6689e87-a087-4b3c-836a-a4cb386ae761/volumes" Nov 02 15:45:22.341429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:22.341347 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" Nov 02 15:45:22.341897 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:22.341350 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p" event={"ID":"bd8f17b6-0b14-475d-b509-cb8121afaef9","Type":"ContainerDied","Data":"5ffc3beefba08a49195d260c328a0a637a2dd07482654c7bf1429211dfd77420"} Nov 02 15:45:22.341897 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:22.341504 5091 scope.go:122] "RemoveContainer" containerID="40831df68390c031c1ce8147b266b6e9f71e3b55f455f4bd8582cb59fd4217b9" Nov 02 15:45:22.505620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:22.505584 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p"] Nov 02 15:45:22.510681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:22.510648 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e3b351-29893905-2jx6p"] Nov 02 15:45:23.347686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:23.347640 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt"] Nov 02 15:45:23.352280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:23.352254 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9ca2cf-8gxvt"] Nov 02 15:45:23.960105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:23.960070 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122a5c3c-d745-469e-8478-d83d6ab6f86a" path="/var/lib/kubelet/pods/122a5c3c-d745-469e-8478-d83d6ab6f86a/volumes" Nov 02 15:45:23.960471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:23.960457 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" path="/var/lib/kubelet/pods/bd8f17b6-0b14-475d-b509-cb8121afaef9/volumes" Nov 02 15:45:27.935612 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:27.935571 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:45:30.009845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:30.009812 5091 scope.go:122] "RemoveContainer" containerID="7520ef3c46013b8beee0f46dccd9d91270473b90fbab015ca40cd6b21b892b80" Nov 02 15:45:30.019804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:30.019785 5091 scope.go:122] "RemoveContainer" containerID="486e66036245ecf21bad1622806271690f3c737efe3e9354edc5efba1a357150" Nov 02 15:45:30.029981 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:30.029952 5091 scope.go:122] "RemoveContainer" containerID="1b9ff189072905d0902a1245478f649e1bb513c7b6b20f46ee1312f30eedd690" Nov 02 15:45:30.039714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:30.039691 5091 scope.go:122] "RemoveContainer" containerID="cc0401aec53c734fc0a980e8d17e9f39784e6fcd716d9050a65ccdaad6f7126a" Nov 02 15:45:31.829611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.829574 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt"] Nov 02 15:45:31.830171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830157 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830172 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830182 5091 cpu_manager.go:414] "containerMap: removing container" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830186 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830198 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830203 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830215 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830219 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830228 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830232 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830238 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830243 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830252 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830257 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830269 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830274 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830280 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830285 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830296 5091 cpu_manager.go:414] "containerMap: removing container" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830301 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830308 5091 cpu_manager.go:414] "containerMap: removing container" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830313 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830318 5091 cpu_manager.go:414] "containerMap: removing container" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830323 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830330 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830334 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830340 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830344 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830356 5091 cpu_manager.go:414] "containerMap: removing container" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830360 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830370 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5cffcab8-f4c0-4d7e-ab2f-06119f06149e" containerName="payload-pre-processing" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830374 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cffcab8-f4c0-4d7e-ab2f-06119f06149e" containerName="payload-pre-processing" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830382 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830386 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830394 5091 cpu_manager.go:414] "containerMap: removing container" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830399 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830405 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830410 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830419 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830423 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830501 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830509 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cffcab8-f4c0-4d7e-ab2f-06119f06149e" containerName="payload-pre-processing" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830529 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830538 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830547 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830553 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830559 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830564 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830569 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830575 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830581 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.830569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830588 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:31.831815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830594 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf821ee-c867-4361-8cd8-649ae39f9cce" containerName="cleanup" Nov 02 15:45:31.831815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830601 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:31.831815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830606 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:31.831815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830610 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:31.831815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.830616 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:31.835299 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.835282 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:31.890338 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.890310 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt"] Nov 02 15:45:31.931978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.931950 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:45:31.932171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.932152 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:45:31.942496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.942465 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:31.942637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.942544 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:31.942637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:31.942606 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxdbg\" (UniqueName: \"kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.043929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.043893 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.044666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.044640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxdbg\" (UniqueName: \"kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.048861 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.048813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.049692 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.049654 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.052462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.052435 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9"] Nov 02 15:45:32.065274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.065246 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.106819 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.106749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxdbg\" (UniqueName: \"kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg\") pod \"maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.155540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.151164 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:32.567385 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:32.567340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt"] Nov 02 15:45:32.572310 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:32.572285 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0eb0f7f_b44e_489c_8872_be6271cdb12d.slice/crio-308e2992756b6e7343a081117d3ef531be4a9b5166de04d41bfa1cbe0b1c5d64 WatchSource:0}: Error finding container 308e2992756b6e7343a081117d3ef531be4a9b5166de04d41bfa1cbe0b1c5d64: Status 404 returned error can't find the container with id 308e2992756b6e7343a081117d3ef531be4a9b5166de04d41bfa1cbe0b1c5d64 Nov 02 15:45:33.421745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:33.421717 5091 generic.go:356] "Generic (PLEG): container finished" podID="f0eb0f7f-b44e-489c-8872-be6271cdb12d" containerID="57efa91190c2de2922e6e3fcbd37abef6ca6e6083bfeeb7e24d7eea5b389dc5c" exitCode=0 Nov 02 15:45:33.422078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:33.421759 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" event={"ID":"f0eb0f7f-b44e-489c-8872-be6271cdb12d","Type":"ContainerDied","Data":"57efa91190c2de2922e6e3fcbd37abef6ca6e6083bfeeb7e24d7eea5b389dc5c"} Nov 02 15:45:33.422078 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:33.421819 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" event={"ID":"f0eb0f7f-b44e-489c-8872-be6271cdb12d","Type":"ContainerStarted","Data":"308e2992756b6e7343a081117d3ef531be4a9b5166de04d41bfa1cbe0b1c5d64"} Nov 02 15:45:34.588618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.588588 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:34.684093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.684024 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca\") pod \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " Nov 02 15:45:34.684211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.684158 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca\") pod \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " Nov 02 15:45:34.684211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.684200 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg\" (UniqueName: \"kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg\") pod \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\" (UID: \"f0eb0f7f-b44e-489c-8872-be6271cdb12d\") " Nov 02 15:45:34.684587 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.684561 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca" pod "f0eb0f7f-b44e-489c-8872-be6271cdb12d" (UID: "f0eb0f7f-b44e-489c-8872-be6271cdb12d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:34.684704 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.684690 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:34.686327 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.686306 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca" pod "f0eb0f7f-b44e-489c-8872-be6271cdb12d" (UID: "f0eb0f7f-b44e-489c-8872-be6271cdb12d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:34.686417 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.686352 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg" pod "f0eb0f7f-b44e-489c-8872-be6271cdb12d" (UID: "f0eb0f7f-b44e-489c-8872-be6271cdb12d"). InnerVolumeSpecName "kube-api-access-jxdbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:34.786212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.786180 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxdbg\" (UniqueName: \"kubernetes.io/projected/f0eb0f7f-b44e-489c-8872-be6271cdb12d-kube-api-access-jxdbg\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:34.786212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:34.786208 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0eb0f7f-b44e-489c-8872-be6271cdb12d-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:35.436004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:35.435971 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" Nov 02 15:45:35.436179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:35.436003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt" event={"ID":"f0eb0f7f-b44e-489c-8872-be6271cdb12d","Type":"ContainerDied","Data":"308e2992756b6e7343a081117d3ef531be4a9b5166de04d41bfa1cbe0b1c5d64"} Nov 02 15:45:35.436179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:35.436077 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="308e2992756b6e7343a081117d3ef531be4a9b5166de04d41bfa1cbe0b1c5d64" Nov 02 15:45:43.402137 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:43.402094 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt"] Nov 02 15:45:44.053135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:44.053077 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx"] Nov 02 15:45:44.056255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:44.056224 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac60516b-fz6qx"] Nov 02 15:45:45.961210 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:45.961173 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa37c4c-62ef-481a-ac39-0d6b897e5eab" path="/var/lib/kubelet/pods/6aa37c4c-62ef-481a-ac39-0d6b897e5eab/volumes" Nov 02 15:45:46.010696 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:46.010661 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp"] Nov 02 15:45:46.057835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:46.057804 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eb6bd4-clwjp"] Nov 02 15:45:47.082068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.082025 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" podUID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" containerName="main" containerID="cri-o://eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616" gracePeriod=45 Nov 02 15:45:47.237171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.237126 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" podUID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" containerName="main" probeResult="failure" output="Get \"https://10.132.0.178:8000/ready\": dial tcp 10.132.0.178:8000: connect: connection refused" Nov 02 15:45:47.365001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.364968 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:45:47.460491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460455 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache\") pod \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " Nov 02 15:45:47.460694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460533 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl\" (UniqueName: \"kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl\") pod \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " Nov 02 15:45:47.460694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460561 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir\") pod \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " Nov 02 15:45:47.460694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460665 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home\") pod \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " Nov 02 15:45:47.460875 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460725 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache" pod "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" (UID: "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:47.460875 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460764 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs\") pod \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " Nov 02 15:45:47.460875 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460818 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm\") pod \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\" (UID: \"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93\") " Nov 02 15:45:47.461044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460874 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir" pod "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" (UID: "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:47.461044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.460891 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home" pod "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" (UID: "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:47.461163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.461144 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:47.461234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.461169 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:47.461234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.461183 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:47.462912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.462887 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm" pod "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" (UID: "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:47.463032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.463001 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs" pod "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" (UID: "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:47.463109 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.463088 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl" pod "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" (UID: "5b6ae906-0e4b-45d5-82d5-79a1f42d8d93"). InnerVolumeSpecName "kube-api-access-ndnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:47.529906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.529866 5091 generic.go:356] "Generic (PLEG): container finished" podID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" containerID="eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616" exitCode=0 Nov 02 15:45:47.529906 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.529910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" Nov 02 15:45:47.530176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.529946 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" event={"ID":"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93","Type":"ContainerDied","Data":"eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616"} Nov 02 15:45:47.530176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.530004 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9" event={"ID":"5b6ae906-0e4b-45d5-82d5-79a1f42d8d93","Type":"ContainerDied","Data":"1493a2b24bc31a596f9755edcb7cd97eb3d2e38dfe926130fd1a5bcc247ffb66"} Nov 02 15:45:47.530176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.530026 5091 scope.go:122] "RemoveContainer" containerID="eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616" Nov 02 15:45:47.541335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.541311 5091 scope.go:122] "RemoveContainer" containerID="eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616" Nov 02 15:45:47.541661 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:47.541640 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616\": container with ID starting with eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616 not found: ID does not exist" containerID="eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616" Nov 02 15:45:47.541734 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.541670 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616"} err="failed to get container status \"eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616\": rpc error: code = NotFound desc = could not find container \"eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616\": container with ID starting with eaab8852e55920260f3ee724df2eef6a217986b74a8e3d783bc0afefa7a91616 not found: ID does not exist" Nov 02 15:45:47.561798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.561767 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:47.561798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.561793 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:47.561952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.561809 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndnzl\" (UniqueName: \"kubernetes.io/projected/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93-kube-api-access-ndnzl\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:47.687722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.687683 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9"] Nov 02 15:45:47.693699 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.693672 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2203c8/test-model-2203c8-kserve-d6b4c787-s4md9"] Nov 02 15:45:47.960815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.960739 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b8e25b-69a6-43a7-a3c2-eff38f71d184" path="/var/lib/kubelet/pods/22b8e25b-69a6-43a7-a3c2-eff38f71d184/volumes" Nov 02 15:45:47.961196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:47.961182 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" path="/var/lib/kubelet/pods/5b6ae906-0e4b-45d5-82d5-79a1f42d8d93/volumes" Nov 02 15:45:48.402959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.402919 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" podUID="5fe06013-f933-4cef-8fb9-0d27846c1a01" containerName="payload-pre-processing" containerID="cri-o://34b7a90942ae5c7459732ff042c9e4d0ed64cfe366581df97317a0d1a0278f47" gracePeriod=25 Nov 02 15:45:48.540797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.540767 5091 generic.go:356] "Generic (PLEG): container finished" podID="5fe06013-f933-4cef-8fb9-0d27846c1a01" containerID="34b7a90942ae5c7459732ff042c9e4d0ed64cfe366581df97317a0d1a0278f47" exitCode=0 Nov 02 15:45:48.540957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.540839 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" event={"ID":"5fe06013-f933-4cef-8fb9-0d27846c1a01","Type":"ContainerDied","Data":"34b7a90942ae5c7459732ff042c9e4d0ed64cfe366581df97317a0d1a0278f47"} Nov 02 15:45:48.672902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.672874 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:45:48.774918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.774884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume\") pod \"5fe06013-f933-4cef-8fb9-0d27846c1a01\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " Nov 02 15:45:48.775096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.774977 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6\" (UniqueName: \"kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6\") pod \"5fe06013-f933-4cef-8fb9-0d27846c1a01\" (UID: \"5fe06013-f933-4cef-8fb9-0d27846c1a01\") " Nov 02 15:45:48.775300 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.775278 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume" pod "5fe06013-f933-4cef-8fb9-0d27846c1a01" (UID: "5fe06013-f933-4cef-8fb9-0d27846c1a01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:48.777274 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.777253 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6" pod "5fe06013-f933-4cef-8fb9-0d27846c1a01" (UID: "5fe06013-f933-4cef-8fb9-0d27846c1a01"). InnerVolumeSpecName "kube-api-access-hd8p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:48.876317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.876279 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fe06013-f933-4cef-8fb9-0d27846c1a01-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:48.876317 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:48.876310 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd8p6\" (UniqueName: \"kubernetes.io/projected/5fe06013-f933-4cef-8fb9-0d27846c1a01-kube-api-access-hd8p6\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:49.549883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:49.549851 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" Nov 02 15:45:49.549883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:49.549863 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt" event={"ID":"5fe06013-f933-4cef-8fb9-0d27846c1a01","Type":"ContainerDied","Data":"df0a9103c5d6553513dadb2cbf9052423a93b2fee4f144615703eaac40aab3f3"} Nov 02 15:45:49.550449 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:49.549932 5091 scope.go:122] "RemoveContainer" containerID="34b7a90942ae5c7459732ff042c9e4d0ed64cfe366581df97317a0d1a0278f47" Nov 02 15:45:49.707129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:49.707096 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt"] Nov 02 15:45:49.754426 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:49.754387 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-fe8ef1-6d57dbf7c9-8zbkt"] Nov 02 15:45:49.972947 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:49.972910 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe06013-f933-4cef-8fb9-0d27846c1a01" path="/var/lib/kubelet/pods/5fe06013-f933-4cef-8fb9-0d27846c1a01/volumes" Nov 02 15:45:53.280146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:53.280111 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4"] Nov 02 15:45:53.280596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:53.280322 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" containerID="cri-o://fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a" gracePeriod=30 Nov 02 15:45:54.188091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188052 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8"] Nov 02 15:45:54.188649 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188633 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:54.188695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188651 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:54.188695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188667 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5fe06013-f933-4cef-8fb9-0d27846c1a01" containerName="payload-pre-processing" Nov 02 15:45:54.188695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188672 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe06013-f933-4cef-8fb9-0d27846c1a01" containerName="payload-pre-processing" Nov 02 15:45:54.188695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188693 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f0eb0f7f-b44e-489c-8872-be6271cdb12d" containerName="revoke-keys" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188698 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0eb0f7f-b44e-489c-8872-be6271cdb12d" containerName="revoke-keys" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188706 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" containerName="main" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188710 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" containerName="main" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188717 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188722 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd8f17b6-0b14-475d-b509-cb8121afaef9" containerName="cleanup" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188797 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe06013-f933-4cef-8fb9-0d27846c1a01" containerName="payload-pre-processing" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188806 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6689e87-a087-4b3c-836a-a4cb386ae761" containerName="cleanup" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188812 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96a77e4-8d39-4ea9-9b7f-9967675d23ff" containerName="cleanup" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188821 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f5bdb7-eb68-4bcf-8d18-e60d3d0a1420" containerName="cleanup" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188827 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0eb0f7f-b44e-489c-8872-be6271cdb12d" containerName="revoke-keys" Nov 02 15:45:54.188829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188833 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a9157f-e263-4a9e-b33f-fa004ab12832" containerName="cleanup" Nov 02 15:45:54.189170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188839 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b6ae906-0e4b-45d5-82d5-79a1f42d8d93" containerName="main" Nov 02 15:45:54.189170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.188852 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="33cc25ee-c935-4bbd-8424-11c2c0c25221" containerName="cleanup" Nov 02 15:45:54.193455 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.193435 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.240387 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.240359 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wts\" (UniqueName: \"kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.240531 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.240411 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.240608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.240568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.243745 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.243724 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:45:54.258555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.247313 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8"] Nov 02 15:45:54.289795 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.289771 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:45:54.341830 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.341802 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wts\" (UniqueName: \"kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.341982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.341881 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.341982 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.341932 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.342732 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.342705 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.344332 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.344303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.400973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.400945 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wts\" (UniqueName: \"kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts\") pod \"maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.507607 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.507528 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:54.707601 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.707563 5091 patch_prober.go:29] interesting pod/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:45:54.707772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.707626 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:45:54.887006 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:54.886972 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8"] Nov 02 15:45:54.891665 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:45:54.891636 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf412dbe2_9fb0_4ad1_94a0_41d47dec0698.slice/crio-e8db4fc14b58c640738fec0b96bd2b80c4a2f36002fe5de797329961f9d2fede WatchSource:0}: Error finding container e8db4fc14b58c640738fec0b96bd2b80c4a2f36002fe5de797329961f9d2fede: Status 404 returned error can't find the container with id e8db4fc14b58c640738fec0b96bd2b80c4a2f36002fe5de797329961f9d2fede Nov 02 15:45:55.598328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:55.598242 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" event={"ID":"f412dbe2-9fb0-4ad1-94a0-41d47dec0698","Type":"ContainerStarted","Data":"be5db6f2f1a3e49d3ffd8003c29becbca1e0e2048b20937e30f4601452c7bae5"} Nov 02 15:45:55.598328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:55.598303 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" event={"ID":"f412dbe2-9fb0-4ad1-94a0-41d47dec0698","Type":"ContainerStarted","Data":"e8db4fc14b58c640738fec0b96bd2b80c4a2f36002fe5de797329961f9d2fede"} Nov 02 15:45:55.710722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:55.710677 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" podStartSLOduration=1.710663559 podStartE2EDuration="1.710663559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:45:55.708444058 +0000 UTC m=+2909.872996600" watchObservedRunningTime="2026-11-02 15:45:55.710663559 +0000 UTC m=+2909.875216095" Nov 02 15:45:56.606626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:56.606591 5091 generic.go:356] "Generic (PLEG): container finished" podID="f412dbe2-9fb0-4ad1-94a0-41d47dec0698" containerID="be5db6f2f1a3e49d3ffd8003c29becbca1e0e2048b20937e30f4601452c7bae5" exitCode=0 Nov 02 15:45:56.607003 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:56.606685 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" event={"ID":"f412dbe2-9fb0-4ad1-94a0-41d47dec0698","Type":"ContainerDied","Data":"be5db6f2f1a3e49d3ffd8003c29becbca1e0e2048b20937e30f4601452c7bae5"} Nov 02 15:45:57.765616 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.765574 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:57.898578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.898539 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca\") pod \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " Nov 02 15:45:57.898783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.898591 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts\" (UniqueName: \"kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts\") pod \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " Nov 02 15:45:57.898783 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.898698 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca\") pod \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\" (UID: \"f412dbe2-9fb0-4ad1-94a0-41d47dec0698\") " Nov 02 15:45:57.898989 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.898955 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca" pod "f412dbe2-9fb0-4ad1-94a0-41d47dec0698" (UID: "f412dbe2-9fb0-4ad1-94a0-41d47dec0698"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:57.899193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.899171 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:57.900801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.900773 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca" pod "f412dbe2-9fb0-4ad1-94a0-41d47dec0698" (UID: "f412dbe2-9fb0-4ad1-94a0-41d47dec0698"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:45:57.900901 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:57.900877 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts" pod "f412dbe2-9fb0-4ad1-94a0-41d47dec0698" (UID: "f412dbe2-9fb0-4ad1-94a0-41d47dec0698"). InnerVolumeSpecName "kube-api-access-t7wts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:57.935301 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:57.935262 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:45:58.000150 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.000119 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wts\" (UniqueName: \"kubernetes.io/projected/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-kube-api-access-t7wts\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.000150 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.000145 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f412dbe2-9fb0-4ad1-94a0-41d47dec0698-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.531251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.531226 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:45:58.606654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.606536 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.606654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.606629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.606996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.606738 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.606996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.606824 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.606996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.606862 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.607206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.607038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.607206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.607109 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.607206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.607157 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.607390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.607278 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data\") pod \"36dc0efb-2f20-413e-9d8b-b26e02769233\" (UID: \"36dc0efb-2f20-413e-9d8b-b26e02769233\") " Nov 02 15:45:58.608197 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.608170 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:58.608389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.608371 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:58.608979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.608895 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:45:58.609498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.609467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:58.609651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.609629 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:58.611471 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.611442 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:58.612863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.612836 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:45:58.612978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.612860 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:45:58.614229 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.614205 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp" pod "36dc0efb-2f20-413e-9d8b-b26e02769233" (UID: "36dc0efb-2f20-413e-9d8b-b26e02769233"). InnerVolumeSpecName "kube-api-access-gk5qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:45:58.621943 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.621919 5091 generic.go:356] "Generic (PLEG): container finished" podID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerID="fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a" exitCode=0 Nov 02 15:45:58.622034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.621968 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" Nov 02 15:45:58.622034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.621998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" event={"ID":"36dc0efb-2f20-413e-9d8b-b26e02769233","Type":"ContainerDied","Data":"fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a"} Nov 02 15:45:58.622120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.622046 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4" event={"ID":"36dc0efb-2f20-413e-9d8b-b26e02769233","Type":"ContainerDied","Data":"73c1ebdc1b98bd6e78657b69cef417bea632fd18d8f0da966d6c62c65151b077"} Nov 02 15:45:58.622120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.622075 5091 scope.go:122] "RemoveContainer" containerID="fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a" Nov 02 15:45:58.623935 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.623915 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" Nov 02 15:45:58.624057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.623962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8" event={"ID":"f412dbe2-9fb0-4ad1-94a0-41d47dec0698","Type":"ContainerDied","Data":"e8db4fc14b58c640738fec0b96bd2b80c4a2f36002fe5de797329961f9d2fede"} Nov 02 15:45:58.624057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.624022 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8db4fc14b58c640738fec0b96bd2b80c4a2f36002fe5de797329961f9d2fede" Nov 02 15:45:58.633342 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.633321 5091 scope.go:122] "RemoveContainer" containerID="3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f" Nov 02 15:45:58.642400 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.642382 5091 scope.go:122] "RemoveContainer" containerID="fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a" Nov 02 15:45:58.642681 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:58.642654 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a\": container with ID starting with fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a not found: ID does not exist" containerID="fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a" Nov 02 15:45:58.642740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.642687 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a"} err="failed to get container status \"fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a\": rpc error: code = NotFound desc = could not find container \"fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a\": container with ID starting with fc90263604f9f3816fdcdd353a694a2d67b04097a2484088a97cfe6b9e1dd87a not found: ID does not exist" Nov 02 15:45:58.642740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.642703 5091 scope.go:122] "RemoveContainer" containerID="3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f" Nov 02 15:45:58.642947 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:45:58.642931 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f\": container with ID starting with 3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f not found: ID does not exist" containerID="3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f" Nov 02 15:45:58.642985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.642951 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f"} err="failed to get container status \"3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f\": rpc error: code = NotFound desc = could not find container \"3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f\": container with ID starting with 3f726df13050d2db7256917900d3c054307fa8d41f51dfc523f4cb5846cffe5f not found: ID does not exist" Nov 02 15:45:58.709489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709453 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709484 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5qp\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-kube-api-access-gk5qp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709502 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709528 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709545 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36dc0efb-2f20-413e-9d8b-b26e02769233-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709556 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709568 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36dc0efb-2f20-413e-9d8b-b26e02769233-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709582 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.709763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.709594 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36dc0efb-2f20-413e-9d8b-b26e02769233-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:45:58.771369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.771333 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4"] Nov 02 15:45:58.785443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:58.781194 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-eb6bd4-openshift-default-5c96c6c447-xl5s4"] Nov 02 15:45:59.960644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:45:59.960609 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" path="/var/lib/kubelet/pods/36dc0efb-2f20-413e-9d8b-b26e02769233/volumes" Nov 02 15:46:05.729329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:05.729290 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9"] Nov 02 15:46:10.729839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:10.729735 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" podUID="7cd9f591-c603-423f-a915-cbf7cd4aabaf" containerName="payload-pre-processing" containerID="cri-o://da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f" gracePeriod=25 Nov 02 15:46:10.994908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:10.994847 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:46:11.059396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.059360 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume\") pod \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " Nov 02 15:46:11.059608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.059459 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4\" (UniqueName: \"kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4\") pod \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\" (UID: \"7cd9f591-c603-423f-a915-cbf7cd4aabaf\") " Nov 02 15:46:11.060021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.059839 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume" pod "7cd9f591-c603-423f-a915-cbf7cd4aabaf" (UID: "7cd9f591-c603-423f-a915-cbf7cd4aabaf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:11.060021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.059968 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7cd9f591-c603-423f-a915-cbf7cd4aabaf-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:11.061770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.061742 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4" pod "7cd9f591-c603-423f-a915-cbf7cd4aabaf" (UID: "7cd9f591-c603-423f-a915-cbf7cd4aabaf"). InnerVolumeSpecName "kube-api-access-pf2q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:11.160662 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.160628 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2q4\" (UniqueName: \"kubernetes.io/projected/7cd9f591-c603-423f-a915-cbf7cd4aabaf-kube-api-access-pf2q4\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:11.717865 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.717830 5091 generic.go:356] "Generic (PLEG): container finished" podID="7cd9f591-c603-423f-a915-cbf7cd4aabaf" containerID="da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f" exitCode=0 Nov 02 15:46:11.718060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.717871 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" Nov 02 15:46:11.718060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.717911 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" event={"ID":"7cd9f591-c603-423f-a915-cbf7cd4aabaf","Type":"ContainerDied","Data":"da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f"} Nov 02 15:46:11.718060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.717964 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9" event={"ID":"7cd9f591-c603-423f-a915-cbf7cd4aabaf","Type":"ContainerDied","Data":"a518aa047a0225d804b4f96a5892defa522b727c1220168a02308c01d7797d3a"} Nov 02 15:46:11.718060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.717986 5091 scope.go:122] "RemoveContainer" containerID="da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f" Nov 02 15:46:11.728359 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.728341 5091 scope.go:122] "RemoveContainer" containerID="da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f" Nov 02 15:46:11.728694 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:46:11.728674 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f\": container with ID starting with da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f not found: ID does not exist" containerID="da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f" Nov 02 15:46:11.728789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.728701 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f"} err="failed to get container status \"da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f\": rpc error: code = NotFound desc = could not find container \"da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f\": container with ID starting with da37e26ba533b4b7b3ce731e90a2889454a66ed6cc5407a4c27afc3793ed335f not found: ID does not exist" Nov 02 15:46:11.875896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.875853 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9"] Nov 02 15:46:11.923109 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.923074 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b1ee37-59bfb769b7-pkcm9"] Nov 02 15:46:11.960379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:11.960343 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd9f591-c603-423f-a915-cbf7cd4aabaf" path="/var/lib/kubelet/pods/7cd9f591-c603-423f-a915-cbf7cd4aabaf/volumes" Nov 02 15:46:15.144429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.144392 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht"] Nov 02 15:46:15.144946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.144929 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7cd9f591-c603-423f-a915-cbf7cd4aabaf" containerName="payload-pre-processing" Nov 02 15:46:15.145011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.144947 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd9f591-c603-423f-a915-cbf7cd4aabaf" containerName="payload-pre-processing" Nov 02 15:46:15.145011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.144962 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f412dbe2-9fb0-4ad1-94a0-41d47dec0698" containerName="revoke-keys" Nov 02 15:46:15.145011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.144968 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f412dbe2-9fb0-4ad1-94a0-41d47dec0698" containerName="revoke-keys" Nov 02 15:46:15.145011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.144996 5091 cpu_manager.go:414] "containerMap: removing container" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" Nov 02 15:46:15.145011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145001 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" Nov 02 15:46:15.145168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145015 5091 cpu_manager.go:414] "containerMap: removing container" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" Nov 02 15:46:15.145168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145019 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" Nov 02 15:46:15.145168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145111 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" Nov 02 15:46:15.145168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145120 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd9f591-c603-423f-a915-cbf7cd4aabaf" containerName="payload-pre-processing" Nov 02 15:46:15.145168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145127 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f412dbe2-9fb0-4ad1-94a0-41d47dec0698" containerName="revoke-keys" Nov 02 15:46:15.145168 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.145138 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="36dc0efb-2f20-413e-9d8b-b26e02769233" containerName="istio-proxy" Nov 02 15:46:15.149710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.149691 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.200125 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.200097 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:46:15.203115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.203082 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht"] Nov 02 15:46:15.206923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.206898 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.207065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.206967 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.207146 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.207058 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r55ck\" (UniqueName: \"kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.246447 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.246417 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:46:15.308810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.308773 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.309027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.308828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.309027 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.308858 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r55ck\" (UniqueName: \"kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.309668 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.309643 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.311378 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.311356 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.367772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.367739 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r55ck\" (UniqueName: \"kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck\") pod \"maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.463172 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.463090 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:15.835959 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:15.835826 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht"] Nov 02 15:46:15.841529 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:46:15.841488 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75a1955_f8b9_4880_b668_05839402aa18.slice/crio-126a9b4bbd41df6ac673620a7ec50c50d6e547aa34e0f4185a7b81b357b33939 WatchSource:0}: Error finding container 126a9b4bbd41df6ac673620a7ec50c50d6e547aa34e0f4185a7b81b357b33939: Status 404 returned error can't find the container with id 126a9b4bbd41df6ac673620a7ec50c50d6e547aa34e0f4185a7b81b357b33939 Nov 02 15:46:16.752949 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:16.752911 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" event={"ID":"d75a1955-f8b9-4880-b668-05839402aa18","Type":"ContainerStarted","Data":"1c856f96adc4590b7545192bc9989e93129d6d68970b5609a8db24e605236ad4"} Nov 02 15:46:16.753391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:16.752974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" event={"ID":"d75a1955-f8b9-4880-b668-05839402aa18","Type":"ContainerStarted","Data":"126a9b4bbd41df6ac673620a7ec50c50d6e547aa34e0f4185a7b81b357b33939"} Nov 02 15:46:16.864707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:16.864655 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" podStartSLOduration=1.864642201 podStartE2EDuration="1.864642201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:16.863918038 +0000 UTC m=+2931.028470582" watchObservedRunningTime="2026-11-02 15:46:16.864642201 +0000 UTC m=+2931.029194741" Nov 02 15:46:17.761669 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:17.761624 5091 generic.go:356] "Generic (PLEG): container finished" podID="d75a1955-f8b9-4880-b668-05839402aa18" containerID="1c856f96adc4590b7545192bc9989e93129d6d68970b5609a8db24e605236ad4" exitCode=0 Nov 02 15:46:17.762127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:17.761682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" event={"ID":"d75a1955-f8b9-4880-b668-05839402aa18","Type":"ContainerDied","Data":"1c856f96adc4590b7545192bc9989e93129d6d68970b5609a8db24e605236ad4"} Nov 02 15:46:18.910177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.910151 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:18.950244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.950216 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca\") pod \"d75a1955-f8b9-4880-b668-05839402aa18\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " Nov 02 15:46:18.950384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.950314 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck\" (UniqueName: \"kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck\") pod \"d75a1955-f8b9-4880-b668-05839402aa18\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " Nov 02 15:46:18.950384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.950348 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca\") pod \"d75a1955-f8b9-4880-b668-05839402aa18\" (UID: \"d75a1955-f8b9-4880-b668-05839402aa18\") " Nov 02 15:46:18.950826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.950794 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca" pod "d75a1955-f8b9-4880-b668-05839402aa18" (UID: "d75a1955-f8b9-4880-b668-05839402aa18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:18.952239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.952205 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca" pod "d75a1955-f8b9-4880-b668-05839402aa18" (UID: "d75a1955-f8b9-4880-b668-05839402aa18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:46:18.952476 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:18.952450 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck" pod "d75a1955-f8b9-4880-b668-05839402aa18" (UID: "d75a1955-f8b9-4880-b668-05839402aa18"). InnerVolumeSpecName "kube-api-access-r55ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:19.053023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:19.052940 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:19.053023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:19.052973 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r55ck\" (UniqueName: \"kubernetes.io/projected/d75a1955-f8b9-4880-b668-05839402aa18-kube-api-access-r55ck\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:19.053023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:19.052983 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d75a1955-f8b9-4880-b668-05839402aa18-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:19.783458 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:19.783419 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" Nov 02 15:46:19.783688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:19.783425 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht" event={"ID":"d75a1955-f8b9-4880-b668-05839402aa18","Type":"ContainerDied","Data":"126a9b4bbd41df6ac673620a7ec50c50d6e547aa34e0f4185a7b81b357b33939"} Nov 02 15:46:19.783688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:19.783636 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126a9b4bbd41df6ac673620a7ec50c50d6e547aa34e0f4185a7b81b357b33939" Nov 02 15:46:23.994626 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:23.994592 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd"] Nov 02 15:46:27.934818 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:46:27.934782 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:46:30.122034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:30.121998 5091 scope.go:122] "RemoveContainer" containerID="b5b2be9ef9372a1fc1729341abdde14326fe350aa6bceac6f9d61922eb5e307e" Nov 02 15:46:30.131988 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:30.131967 5091 scope.go:122] "RemoveContainer" containerID="2fd61e185d3d2aca9c51619e47852ac2362f546e7d7f8e81f65072789bc6d270" Nov 02 15:46:35.437554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:35.437499 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7"] Nov 02 15:46:35.486182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:35.486146 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf"] Nov 02 15:46:35.886121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:35.886082 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" containerID="cri-o://bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c" gracePeriod=30 Nov 02 15:46:35.886121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:35.886111 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" containerID="cri-o://d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07" gracePeriod=30 Nov 02 15:46:35.939098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:35.939066 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-vwzpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:46:35.939218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:35.939148 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:46:37.598782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:37.598742 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-bf9z7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:46:37.599189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:37.598795 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:46:39.014891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.014834 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" podUID="fa350c5f-bf5c-4998-af63-41ab18d69f72" containerName="main" containerID="cri-o://f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20" gracePeriod=45 Nov 02 15:46:39.292390 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.292366 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:46:39.390905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.390878 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm\") pod \"fa350c5f-bf5c-4998-af63-41ab18d69f72\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " Nov 02 15:46:39.391058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.390966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache\") pod \"fa350c5f-bf5c-4998-af63-41ab18d69f72\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " Nov 02 15:46:39.391058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs\") pod \"fa350c5f-bf5c-4998-af63-41ab18d69f72\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " Nov 02 15:46:39.391141 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391090 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home\") pod \"fa350c5f-bf5c-4998-af63-41ab18d69f72\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " Nov 02 15:46:39.391141 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391109 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir\") pod \"fa350c5f-bf5c-4998-af63-41ab18d69f72\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " Nov 02 15:46:39.391231 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391169 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds\" (UniqueName: \"kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds\") pod \"fa350c5f-bf5c-4998-af63-41ab18d69f72\" (UID: \"fa350c5f-bf5c-4998-af63-41ab18d69f72\") " Nov 02 15:46:39.391289 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391223 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache" pod "fa350c5f-bf5c-4998-af63-41ab18d69f72" (UID: "fa350c5f-bf5c-4998-af63-41ab18d69f72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:39.391383 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391362 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home" pod "fa350c5f-bf5c-4998-af63-41ab18d69f72" (UID: "fa350c5f-bf5c-4998-af63-41ab18d69f72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:39.391451 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391385 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir" pod "fa350c5f-bf5c-4998-af63-41ab18d69f72" (UID: "fa350c5f-bf5c-4998-af63-41ab18d69f72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:39.391716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391693 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:39.391716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391714 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:39.391924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.391724 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:39.393351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.393318 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm" pod "fa350c5f-bf5c-4998-af63-41ab18d69f72" (UID: "fa350c5f-bf5c-4998-af63-41ab18d69f72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:39.393469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.393383 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs" pod "fa350c5f-bf5c-4998-af63-41ab18d69f72" (UID: "fa350c5f-bf5c-4998-af63-41ab18d69f72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:46:39.393469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.393441 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds" pod "fa350c5f-bf5c-4998-af63-41ab18d69f72" (UID: "fa350c5f-bf5c-4998-af63-41ab18d69f72"). InnerVolumeSpecName "kube-api-access-qfqds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:39.492409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.492374 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa350c5f-bf5c-4998-af63-41ab18d69f72-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:39.492409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.492402 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqds\" (UniqueName: \"kubernetes.io/projected/fa350c5f-bf5c-4998-af63-41ab18d69f72-kube-api-access-qfqds\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:39.492409 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.492414 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa350c5f-bf5c-4998-af63-41ab18d69f72-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:39.911068 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.911031 5091 generic.go:356] "Generic (PLEG): container finished" podID="fa350c5f-bf5c-4998-af63-41ab18d69f72" containerID="f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20" exitCode=0 Nov 02 15:46:39.911247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.911079 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" Nov 02 15:46:39.911247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.911081 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" event={"ID":"fa350c5f-bf5c-4998-af63-41ab18d69f72","Type":"ContainerDied","Data":"f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20"} Nov 02 15:46:39.911247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.911121 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd" event={"ID":"fa350c5f-bf5c-4998-af63-41ab18d69f72","Type":"ContainerDied","Data":"49fc1ca4e097c27a10a9caa48d21eaf57e194aae716a8ccef130147edf9106e2"} Nov 02 15:46:39.911247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.911149 5091 scope.go:122] "RemoveContainer" containerID="f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20" Nov 02 15:46:39.921319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.921303 5091 scope.go:122] "RemoveContainer" containerID="f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20" Nov 02 15:46:39.921586 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:46:39.921566 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20\": container with ID starting with f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20 not found: ID does not exist" containerID="f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20" Nov 02 15:46:39.921644 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:39.921594 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20"} err="failed to get container status \"f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20\": rpc error: code = NotFound desc = could not find container \"f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20\": container with ID starting with f76fb6d9212c50a8472901ec162647a8996522d6c37263280ef567b22a388a20 not found: ID does not exist" Nov 02 15:46:40.072714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:40.072676 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd"] Nov 02 15:46:40.117915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:40.117879 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f6b245/test-model-f6b245-kserve-56dbb8ddd9-99pzd"] Nov 02 15:46:41.193453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.193368 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:46:41.227282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.227252 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:46:41.317424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317396 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.317598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317448 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.317598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317477 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.317598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317532 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.317598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317557 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317619 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317674 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317724 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317765 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317752 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317799 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.317841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317825 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317857 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317861 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317963 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.317998 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318039 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318064 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo\") pod \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\" (UID: \"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318098 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo\") pod \"25af89a0-678e-4c36-99dd-54000948f42e\" (UID: \"25af89a0-678e-4c36-99dd-54000948f42e\") " Nov 02 15:46:41.318216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318201 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318317 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318384 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318538 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318739 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318854 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318877 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318892 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.318907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318906 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.320475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318919 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.320475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318946 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.320475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318963 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25af89a0-678e-4c36-99dd-54000948f42e-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.320475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318978 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.320475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.318992 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.320791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.320621 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "kube-api-access-98t4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:41.320791 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.320725 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.321076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.321008 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:41.321187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.321120 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:41.321233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.321204 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "kube-api-access-dkchj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:41.321404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.321388 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy" pod "25af89a0-678e-4c36-99dd-54000948f42e" (UID: "25af89a0-678e-4c36-99dd-54000948f42e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:46:41.321508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.321490 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:46:41.322287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.322272 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo" pod "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" (UID: "4b85cbdb-53a9-4259-9c5c-b1c14e672e5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:46:41.419792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419766 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98t4m\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-kube-api-access-98t4m\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419792 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419791 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419803 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419812 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkchj\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-kube-api-access-dkchj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419821 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25af89a0-678e-4c36-99dd-54000948f42e-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419830 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25af89a0-678e-4c36-99dd-54000948f42e-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419839 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419848 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.419980 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.419856 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25af89a0-678e-4c36-99dd-54000948f42e-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:46:41.930976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.930944 5091 generic.go:356] "Generic (PLEG): container finished" podID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerID="d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07" exitCode=0 Nov 02 15:46:41.931127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.930993 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" Nov 02 15:46:41.931127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.931027 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" event={"ID":"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d","Type":"ContainerDied","Data":"d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07"} Nov 02 15:46:41.931127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.931088 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7" event={"ID":"4b85cbdb-53a9-4259-9c5c-b1c14e672e5d","Type":"ContainerDied","Data":"0aceb0d2b98615bdd5f13a6c1467b648d7b51cc8ec69f04a197a09a7ab7f4722"} Nov 02 15:46:41.931127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.931119 5091 scope.go:122] "RemoveContainer" containerID="d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07" Nov 02 15:46:41.932691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.932665 5091 generic.go:356] "Generic (PLEG): container finished" podID="25af89a0-678e-4c36-99dd-54000948f42e" containerID="bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c" exitCode=0 Nov 02 15:46:41.932814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.932710 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" event={"ID":"25af89a0-678e-4c36-99dd-54000948f42e","Type":"ContainerDied","Data":"bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c"} Nov 02 15:46:41.932814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.932724 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" Nov 02 15:46:41.932814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.932748 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf" event={"ID":"25af89a0-678e-4c36-99dd-54000948f42e","Type":"ContainerDied","Data":"2345eabda4d2987665cd1f728c302622db032b6ecd803028cd0ebd9d8700dcb4"} Nov 02 15:46:41.943402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.943381 5091 scope.go:122] "RemoveContainer" containerID="d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07" Nov 02 15:46:41.943690 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:46:41.943666 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07\": container with ID starting with d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07 not found: ID does not exist" containerID="d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07" Nov 02 15:46:41.943776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.943701 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07"} err="failed to get container status \"d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07\": rpc error: code = NotFound desc = could not find container \"d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07\": container with ID starting with d09714888675ccb026c6c850168417618f654e0212ed4329646e809ad4156b07 not found: ID does not exist" Nov 02 15:46:41.943776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.943724 5091 scope.go:122] "RemoveContainer" containerID="bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c" Nov 02 15:46:41.952667 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.952651 5091 scope.go:122] "RemoveContainer" containerID="bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c" Nov 02 15:46:41.952939 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:46:41.952919 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c\": container with ID starting with bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c not found: ID does not exist" containerID="bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c" Nov 02 15:46:41.953001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.952945 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c"} err="failed to get container status \"bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c\": rpc error: code = NotFound desc = could not find container \"bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c\": container with ID starting with bb9eb1fc7839610fbe10ee94537730ae7e482accb6e041422d3321a662dc001c not found: ID does not exist" Nov 02 15:46:41.962941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:41.962919 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa350c5f-bf5c-4998-af63-41ab18d69f72" path="/var/lib/kubelet/pods/fa350c5f-bf5c-4998-af63-41ab18d69f72/volumes" Nov 02 15:46:42.138954 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:42.138913 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7"] Nov 02 15:46:42.150763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:42.150725 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-bf9z7"] Nov 02 15:46:42.254654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:42.254569 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf"] Nov 02 15:46:42.257254 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:42.257228 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-vwzpf"] Nov 02 15:46:43.961452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:43.961416 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25af89a0-678e-4c36-99dd-54000948f42e" path="/var/lib/kubelet/pods/25af89a0-678e-4c36-99dd-54000948f42e/volumes" Nov 02 15:46:43.962038 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:43.962021 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" path="/var/lib/kubelet/pods/4b85cbdb-53a9-4259-9c5c-b1c14e672e5d/volumes" Nov 02 15:46:56.505142 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:56.505108 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm"] Nov 02 15:46:56.505845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:56.505359 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" containerID="cri-o://cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49" gracePeriod=30 Nov 02 15:46:57.655532 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.655490 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh"] Nov 02 15:46:57.656055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656039 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d75a1955-f8b9-4880-b668-05839402aa18" containerName="revoke-keys" Nov 02 15:46:57.656111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656057 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75a1955-f8b9-4880-b668-05839402aa18" containerName="revoke-keys" Nov 02 15:46:57.656111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656081 5091 cpu_manager.go:414] "containerMap: removing container" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" Nov 02 15:46:57.656111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656086 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" Nov 02 15:46:57.656111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656099 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fa350c5f-bf5c-4998-af63-41ab18d69f72" containerName="main" Nov 02 15:46:57.656111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656104 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa350c5f-bf5c-4998-af63-41ab18d69f72" containerName="main" Nov 02 15:46:57.656266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656138 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" Nov 02 15:46:57.656266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656144 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" Nov 02 15:46:57.656266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656239 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b85cbdb-53a9-4259-9c5c-b1c14e672e5d" containerName="istio-proxy" Nov 02 15:46:57.656266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656252 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75a1955-f8b9-4880-b668-05839402aa18" containerName="revoke-keys" Nov 02 15:46:57.656266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656264 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="25af89a0-678e-4c36-99dd-54000948f42e" containerName="istio-proxy" Nov 02 15:46:57.656424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.656272 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa350c5f-bf5c-4998-af63-41ab18d69f72" containerName="main" Nov 02 15:46:57.659818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.659799 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.714401 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.714374 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh"] Nov 02 15:46:57.757039 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.757017 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:46:57.757200 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.757131 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:46:57.805265 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.805241 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.805410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.805288 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.805472 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.805447 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcxn\" (UniqueName: \"kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.906603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.906504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.906603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.906582 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.906814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.906658 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcxn\" (UniqueName: \"kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.907366 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.907335 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.909181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.909147 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.935889 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:46:57.935853 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:46:57.971011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.969077 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcxn\" (UniqueName: \"kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn\") pod \"maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:57.977275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:57.977241 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:46:58.353576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:58.353471 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh"] Nov 02 15:46:58.357182 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:46:58.357142 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87793f1e_4daf_45b3_beee_969b101950c3.slice/crio-9b41d32c05f734d3651c07b135d5e9e634ba960e7ce24d1f7843b30c8e320a92 WatchSource:0}: Error finding container 9b41d32c05f734d3651c07b135d5e9e634ba960e7ce24d1f7843b30c8e320a92: Status 404 returned error can't find the container with id 9b41d32c05f734d3651c07b135d5e9e634ba960e7ce24d1f7843b30c8e320a92 Nov 02 15:46:59.058687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:59.058607 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" event={"ID":"87793f1e-4daf-45b3-beee-969b101950c3","Type":"ContainerStarted","Data":"a813f5ccc42aa5877f69b9635ccf890b24d0b7d43071b2ebd5e537be87f4b59d"} Nov 02 15:46:59.058687 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:59.058663 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" event={"ID":"87793f1e-4daf-45b3-beee-969b101950c3","Type":"ContainerStarted","Data":"9b41d32c05f734d3651c07b135d5e9e634ba960e7ce24d1f7843b30c8e320a92"} Nov 02 15:46:59.171838 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:46:59.171784 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" podStartSLOduration=2.171770323 podStartE2EDuration="2.171770323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:46:59.168465754 +0000 UTC m=+2973.333018292" watchObservedRunningTime="2026-11-02 15:46:59.171770323 +0000 UTC m=+2973.336322863" Nov 02 15:47:00.066246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:00.066202 5091 generic.go:356] "Generic (PLEG): container finished" podID="87793f1e-4daf-45b3-beee-969b101950c3" containerID="a813f5ccc42aa5877f69b9635ccf890b24d0b7d43071b2ebd5e537be87f4b59d" exitCode=0 Nov 02 15:47:00.066786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:00.066296 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" event={"ID":"87793f1e-4daf-45b3-beee-969b101950c3","Type":"ContainerDied","Data":"a813f5ccc42aa5877f69b9635ccf890b24d0b7d43071b2ebd5e537be87f4b59d"} Nov 02 15:47:01.213879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.213855 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:47:01.344306 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.344213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn\" (UniqueName: \"kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn\") pod \"87793f1e-4daf-45b3-beee-969b101950c3\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " Nov 02 15:47:01.344489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.344403 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca\") pod \"87793f1e-4daf-45b3-beee-969b101950c3\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " Nov 02 15:47:01.344489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.344469 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca\") pod \"87793f1e-4daf-45b3-beee-969b101950c3\" (UID: \"87793f1e-4daf-45b3-beee-969b101950c3\") " Nov 02 15:47:01.345050 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.345005 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca" pod "87793f1e-4daf-45b3-beee-969b101950c3" (UID: "87793f1e-4daf-45b3-beee-969b101950c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:01.347061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.347029 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca" pod "87793f1e-4daf-45b3-beee-969b101950c3" (UID: "87793f1e-4daf-45b3-beee-969b101950c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:01.349045 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.349017 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn" pod "87793f1e-4daf-45b3-beee-969b101950c3" (UID: "87793f1e-4daf-45b3-beee-969b101950c3"). InnerVolumeSpecName "kube-api-access-zdcxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:01.445944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.445902 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdcxn\" (UniqueName: \"kubernetes.io/projected/87793f1e-4daf-45b3-beee-969b101950c3-kube-api-access-zdcxn\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.445944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.445944 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.446138 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.445963 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87793f1e-4daf-45b3-beee-969b101950c3-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.753253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.753222 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:47:01.849895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.849859 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.849953 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850011 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850100 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:01.850187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850144 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850173 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850193 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850220 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850285 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850353 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert\") pod \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\" (UID: \"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1\") " Nov 02 15:47:01.850508 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850468 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:01.850656 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850505 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:01.850720 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850654 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:01.851107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850947 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.851107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850966 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:01.851107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.850973 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.851107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.851010 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.851107 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.851027 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.852402 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.852377 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:47:01.852634 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.852617 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:01.852706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.852625 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:01.852870 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.852853 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5" pod "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" (UID: "bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1"). InnerVolumeSpecName "kube-api-access-hb6c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:01.952064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.951983 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.952064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.952009 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.952064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.952021 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6c5\" (UniqueName: \"kubernetes.io/projected/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-kube-api-access-hb6c5\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.952064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.952031 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:01.952064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:01.952045 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:02.047365 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.047333 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws"] Nov 02 15:47:02.081683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.081650 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" Nov 02 15:47:02.081850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.081685 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh" event={"ID":"87793f1e-4daf-45b3-beee-969b101950c3","Type":"ContainerDied","Data":"9b41d32c05f734d3651c07b135d5e9e634ba960e7ce24d1f7843b30c8e320a92"} Nov 02 15:47:02.081850 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.081734 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b41d32c05f734d3651c07b135d5e9e634ba960e7ce24d1f7843b30c8e320a92" Nov 02 15:47:02.083832 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.083797 5091 generic.go:356] "Generic (PLEG): container finished" podID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerID="cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49" exitCode=0 Nov 02 15:47:02.083964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.083843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" event={"ID":"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1","Type":"ContainerDied","Data":"cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49"} Nov 02 15:47:02.083964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.083881 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" event={"ID":"bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1","Type":"ContainerDied","Data":"25ba42b9a2fbe7ab55559178c208cab6b7f96bab9122d814d62c8b048780f151"} Nov 02 15:47:02.083964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.083910 5091 scope.go:122] "RemoveContainer" containerID="cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49" Nov 02 15:47:02.084143 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.084061 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm" Nov 02 15:47:02.094234 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.094216 5091 scope.go:122] "RemoveContainer" containerID="55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62" Nov 02 15:47:02.102946 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.102926 5091 scope.go:122] "RemoveContainer" containerID="cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49" Nov 02 15:47:02.103223 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:02.103202 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49\": container with ID starting with cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49 not found: ID does not exist" containerID="cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49" Nov 02 15:47:02.103281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.103230 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49"} err="failed to get container status \"cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49\": rpc error: code = NotFound desc = could not find container \"cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49\": container with ID starting with cb4fe54a3f7b2abc1d90adaf5dc87989122a0f4744f2c67be1339399b72aaa49 not found: ID does not exist" Nov 02 15:47:02.103281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.103246 5091 scope.go:122] "RemoveContainer" containerID="55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62" Nov 02 15:47:02.103503 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:02.103486 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62\": container with ID starting with 55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62 not found: ID does not exist" containerID="55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62" Nov 02 15:47:02.103566 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.103510 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62"} err="failed to get container status \"55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62\": rpc error: code = NotFound desc = could not find container \"55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62\": container with ID starting with 55cebaa4ce85b1fa6a94540e80204ce6d53a3e58fabf5e5685b8fde600085f62 not found: ID does not exist" Nov 02 15:47:02.268356 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.268281 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm"] Nov 02 15:47:02.275492 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:02.275462 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e3b351-openshift-default-7d9fb8f549-5cmmm"] Nov 02 15:47:03.960573 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:03.960538 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" path="/var/lib/kubelet/pods/bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1/volumes" Nov 02 15:47:09.048911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:09.048874 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs"] Nov 02 15:47:09.096397 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:09.096353 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq"] Nov 02 15:47:09.098924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:09.098885 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22b2afe9-4zjnq"] Nov 02 15:47:09.101052 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:09.101023 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f121d918-sc9hs"] Nov 02 15:47:09.963775 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:09.963739 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b33bc4-9dea-4e40-874a-955223c7c750" path="/var/lib/kubelet/pods/55b33bc4-9dea-4e40-874a-955223c7c750/volumes" Nov 02 15:47:09.964357 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:09.964339 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d174e335-7f8d-488b-98e6-36a8f033417a" path="/var/lib/kubelet/pods/d174e335-7f8d-488b-98e6-36a8f033417a/volumes" Nov 02 15:47:12.056042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:12.055996 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht"] Nov 02 15:47:12.103034 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:12.102988 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2203c8-2s6ht"] Nov 02 15:47:13.962660 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:13.962620 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75a1955-f8b9-4880-b668-05839402aa18" path="/var/lib/kubelet/pods/d75a1955-f8b9-4880-b668-05839402aa18/volumes" Nov 02 15:47:17.068693 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.068648 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" containerID="cri-o://e8e7d667ea8b37c6415ff672c9f3ae47d95ad15e8f3108f41eed768a0ca569af" gracePeriod=45 Nov 02 15:47:17.187011 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.186979 5091 generic.go:356] "Generic (PLEG): container finished" podID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerID="e8e7d667ea8b37c6415ff672c9f3ae47d95ad15e8f3108f41eed768a0ca569af" exitCode=0 Nov 02 15:47:17.187186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.187048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" event={"ID":"a76cfa6b-9055-41e3-a90f-b8f220eb5595","Type":"ContainerDied","Data":"e8e7d667ea8b37c6415ff672c9f3ae47d95ad15e8f3108f41eed768a0ca569af"} Nov 02 15:47:17.346931 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.346896 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:47:17.416042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.415994 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir\") pod \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " Nov 02 15:47:17.416239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416050 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs\" (UniqueName: \"kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs\") pod \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " Nov 02 15:47:17.416239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416149 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm\") pod \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " Nov 02 15:47:17.416239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416176 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home\") pod \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " Nov 02 15:47:17.416239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416211 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache\") pod \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " Nov 02 15:47:17.416480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416344 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir" pod "a76cfa6b-9055-41e3-a90f-b8f220eb5595" (UID: "a76cfa6b-9055-41e3-a90f-b8f220eb5595"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.416480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416378 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5\" (UniqueName: \"kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5\") pod \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\" (UID: \"a76cfa6b-9055-41e3-a90f-b8f220eb5595\") " Nov 02 15:47:17.416647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416486 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home" pod "a76cfa6b-9055-41e3-a90f-b8f220eb5595" (UID: "a76cfa6b-9055-41e3-a90f-b8f220eb5595"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.416647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416571 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache" pod "a76cfa6b-9055-41e3-a90f-b8f220eb5595" (UID: "a76cfa6b-9055-41e3-a90f-b8f220eb5595"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.416832 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416816 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.416871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416834 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.416871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.416844 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.418714 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.418683 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5" pod "a76cfa6b-9055-41e3-a90f-b8f220eb5595" (UID: "a76cfa6b-9055-41e3-a90f-b8f220eb5595"). InnerVolumeSpecName "kube-api-access-tw4g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:17.418922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.418899 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs" pod "a76cfa6b-9055-41e3-a90f-b8f220eb5595" (UID: "a76cfa6b-9055-41e3-a90f-b8f220eb5595"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:17.419010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.418993 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm" pod "a76cfa6b-9055-41e3-a90f-b8f220eb5595" (UID: "a76cfa6b-9055-41e3-a90f-b8f220eb5595"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:17.518481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.518430 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw4g5\" (UniqueName: \"kubernetes.io/projected/a76cfa6b-9055-41e3-a90f-b8f220eb5595-kube-api-access-tw4g5\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.518481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.518475 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76cfa6b-9055-41e3-a90f-b8f220eb5595-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:17.518767 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:17.518491 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76cfa6b-9055-41e3-a90f-b8f220eb5595-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:18.195004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.194969 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" Nov 02 15:47:18.195461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.195005 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws" event={"ID":"a76cfa6b-9055-41e3-a90f-b8f220eb5595","Type":"ContainerDied","Data":"6e34aa8eb9a90a0c1bf545755aa1cea13073a7d897105b0feb75b659b56fc110"} Nov 02 15:47:18.195461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.195056 5091 scope.go:122] "RemoveContainer" containerID="e8e7d667ea8b37c6415ff672c9f3ae47d95ad15e8f3108f41eed768a0ca569af" Nov 02 15:47:18.353232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.353120 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws"] Nov 02 15:47:18.400480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.400441 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-e98976/distinct-2-w6-e98976-kserve-c48c84d96-n8gws"] Nov 02 15:47:18.752029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.751989 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt"] Nov 02 15:47:18.754727 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:18.754695 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-fe8ef1-6zjgt"] Nov 02 15:47:19.961695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:19.961657 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" path="/var/lib/kubelet/pods/a76cfa6b-9055-41e3-a90f-b8f220eb5595/volumes" Nov 02 15:47:19.962148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:19.962133 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0eb0f7f-b44e-489c-8872-be6271cdb12d" path="/var/lib/kubelet/pods/f0eb0f7f-b44e-489c-8872-be6271cdb12d/volumes" Nov 02 15:47:20.451189 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451147 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2"] Nov 02 15:47:20.451804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451784 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" Nov 02 15:47:20.451853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451806 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" Nov 02 15:47:20.451853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451826 5091 cpu_manager.go:414] "containerMap: removing container" podUID="87793f1e-4daf-45b3-beee-969b101950c3" containerName="revoke-keys" Nov 02 15:47:20.451853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451831 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87793f1e-4daf-45b3-beee-969b101950c3" containerName="revoke-keys" Nov 02 15:47:20.451853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451840 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" Nov 02 15:47:20.451853 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451845 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" Nov 02 15:47:20.452029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451926 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" Nov 02 15:47:20.452029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451938 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="87793f1e-4daf-45b3-beee-969b101950c3" containerName="revoke-keys" Nov 02 15:47:20.452029 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.451945 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76cfa6b-9055-41e3-a90f-b8f220eb5595" containerName="main" Nov 02 15:47:20.456843 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.456815 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.513888 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.513852 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2"] Nov 02 15:47:20.556606 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.556556 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.556814 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.556691 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.556856 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.556845 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvhgw\" (UniqueName: \"kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.597026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.596990 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:20.597219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.597035 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:47:20.657899 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.657851 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.658106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.657939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.658106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.658067 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvhgw\" (UniqueName: \"kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.658647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.658625 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.660607 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.660574 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.717487 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.717401 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvhgw\" (UniqueName: \"kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw\") pod \"maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:20.770966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:20.770931 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:21.151864 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:21.151825 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2"] Nov 02 15:47:21.156343 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:21.156305 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0be734_1c4a_4739_bfdb_821271658242.slice/crio-7ae89d4840e1cf1839373b6a73247ad7dd62d25e9ca055b178aa937276346d64 WatchSource:0}: Error finding container 7ae89d4840e1cf1839373b6a73247ad7dd62d25e9ca055b178aa937276346d64: Status 404 returned error can't find the container with id 7ae89d4840e1cf1839373b6a73247ad7dd62d25e9ca055b178aa937276346d64 Nov 02 15:47:21.215575 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:21.215526 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" event={"ID":"bb0be734-1c4a-4739-bfdb-821271658242","Type":"ContainerStarted","Data":"7ae89d4840e1cf1839373b6a73247ad7dd62d25e9ca055b178aa937276346d64"} Nov 02 15:47:22.223355 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:22.223321 5091 generic.go:356] "Generic (PLEG): container finished" podID="bb0be734-1c4a-4739-bfdb-821271658242" containerID="9244d67e383d8e2d65f50bee433ab1809a5b478bedba69eb1ee85a306496256a" exitCode=0 Nov 02 15:47:22.223802 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:22.223412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" event={"ID":"bb0be734-1c4a-4739-bfdb-821271658242","Type":"ContainerDied","Data":"9244d67e383d8e2d65f50bee433ab1809a5b478bedba69eb1ee85a306496256a"} Nov 02 15:47:23.092827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.092786 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6"] Nov 02 15:47:23.093032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.093001 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" podUID="b47059a3-a218-4d1c-9e27-5d4a5d968ecc" containerName="maas-api" containerID="cri-o://b2e9b56ed92856517c773edbdfcd4fd2fbedc51995323e62844ae2a41732d8d6" gracePeriod=30 Nov 02 15:47:23.233459 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.233416 5091 generic.go:356] "Generic (PLEG): container finished" podID="b47059a3-a218-4d1c-9e27-5d4a5d968ecc" containerID="b2e9b56ed92856517c773edbdfcd4fd2fbedc51995323e62844ae2a41732d8d6" exitCode=0 Nov 02 15:47:23.233938 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.233489 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" event={"ID":"b47059a3-a218-4d1c-9e27-5d4a5d968ecc","Type":"ContainerDied","Data":"b2e9b56ed92856517c773edbdfcd4fd2fbedc51995323e62844ae2a41732d8d6"} Nov 02 15:47:23.432225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.432190 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:47:23.493770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.493731 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:23.498485 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.498451 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59\" (UniqueName: \"kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59\") pod \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " Nov 02 15:47:23.498716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.498587 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs\") pod \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " Nov 02 15:47:23.498716 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.498620 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\") pod \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " Nov 02 15:47:23.498852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.498769 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca\") pod \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\" (UID: \"b47059a3-a218-4d1c-9e27-5d4a5d968ecc\") " Nov 02 15:47:23.499284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.499219 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca" pod "b47059a3-a218-4d1c-9e27-5d4a5d968ecc" (UID: "b47059a3-a218-4d1c-9e27-5d4a5d968ecc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:23.501393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.501357 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs" pod "b47059a3-a218-4d1c-9e27-5d4a5d968ecc" (UID: "b47059a3-a218-4d1c-9e27-5d4a5d968ecc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:23.501393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.501375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls" pod "b47059a3-a218-4d1c-9e27-5d4a5d968ecc" (UID: "b47059a3-a218-4d1c-9e27-5d4a5d968ecc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:23.501651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.501418 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59" pod "b47059a3-a218-4d1c-9e27-5d4a5d968ecc" (UID: "b47059a3-a218-4d1c-9e27-5d4a5d968ecc"). InnerVolumeSpecName "kube-api-access-hbl59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:23.599739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.599706 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca\") pod \"bb0be734-1c4a-4739-bfdb-821271658242\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " Nov 02 15:47:23.599937 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.599919 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca\") pod \"bb0be734-1c4a-4739-bfdb-821271658242\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " Nov 02 15:47:23.600018 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.599974 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw\" (UniqueName: \"kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw\") pod \"bb0be734-1c4a-4739-bfdb-821271658242\" (UID: \"bb0be734-1c4a-4739-bfdb-821271658242\") " Nov 02 15:47:23.600339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.600309 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca" pod "bb0be734-1c4a-4739-bfdb-821271658242" (UID: "bb0be734-1c4a-4739-bfdb-821271658242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:23.600469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.600339 5091 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-openshift-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:23.600469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.600359 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbl59\" (UniqueName: \"kubernetes.io/projected/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-kube-api-access-hbl59\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:23.600469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.600369 5091 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-metrics-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:23.600469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.600379 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47059a3-a218-4d1c-9e27-5d4a5d968ecc-maas-api-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:23.602328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.602299 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw" pod "bb0be734-1c4a-4739-bfdb-821271658242" (UID: "bb0be734-1c4a-4739-bfdb-821271658242"). InnerVolumeSpecName "kube-api-access-mvhgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:23.602498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.602475 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca" pod "bb0be734-1c4a-4739-bfdb-821271658242" (UID: "bb0be734-1c4a-4739-bfdb-821271658242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:23.701893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.701777 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:23.701893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.701822 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvhgw\" (UniqueName: \"kubernetes.io/projected/bb0be734-1c4a-4739-bfdb-821271658242-kube-api-access-mvhgw\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:23.701893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:23.701832 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb0be734-1c4a-4739-bfdb-821271658242-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:24.244470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.244375 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" Nov 02 15:47:24.244470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.244393 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6" event={"ID":"b47059a3-a218-4d1c-9e27-5d4a5d968ecc","Type":"ContainerDied","Data":"1cfb73cc9f965f70152a72664208b3e2224789c6acd82cf9cdd75cd4c688b3ac"} Nov 02 15:47:24.245043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.244477 5091 scope.go:122] "RemoveContainer" containerID="b2e9b56ed92856517c773edbdfcd4fd2fbedc51995323e62844ae2a41732d8d6" Nov 02 15:47:24.246496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.246266 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" event={"ID":"bb0be734-1c4a-4739-bfdb-821271658242","Type":"ContainerDied","Data":"7ae89d4840e1cf1839373b6a73247ad7dd62d25e9ca055b178aa937276346d64"} Nov 02 15:47:24.246496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.246322 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ae89d4840e1cf1839373b6a73247ad7dd62d25e9ca055b178aa937276346d64" Nov 02 15:47:24.246496 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.246387 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2" Nov 02 15:47:24.453106 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.453063 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6"] Nov 02 15:47:24.461812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:24.461769 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b1ee37-6b465f7bd7-6trj6"] Nov 02 15:47:25.480432 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:25.480392 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8"] Nov 02 15:47:25.483618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:25.483591 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b1ee37-mcnn8"] Nov 02 15:47:25.960923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:25.960881 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47059a3-a218-4d1c-9e27-5d4a5d968ecc" path="/var/lib/kubelet/pods/b47059a3-a218-4d1c-9e27-5d4a5d968ecc/volumes" Nov 02 15:47:25.961569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:25.961507 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f412dbe2-9fb0-4ad1-94a0-41d47dec0698" path="/var/lib/kubelet/pods/f412dbe2-9fb0-4ad1-94a0-41d47dec0698/volumes" Nov 02 15:47:27.406075 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406037 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms"] Nov 02 15:47:27.406666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406549 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b47059a3-a218-4d1c-9e27-5d4a5d968ecc" containerName="maas-api" Nov 02 15:47:27.406666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406563 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47059a3-a218-4d1c-9e27-5d4a5d968ecc" containerName="maas-api" Nov 02 15:47:27.406666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406585 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" Nov 02 15:47:27.406666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406591 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" Nov 02 15:47:27.406666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406598 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bb0be734-1c4a-4739-bfdb-821271658242" containerName="revoke-keys" Nov 02 15:47:27.406666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406603 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0be734-1c4a-4739-bfdb-821271658242" containerName="revoke-keys" Nov 02 15:47:27.406978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406684 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbbf3604-a959-4e7b-879c-fe8f3d4d2cb1" containerName="istio-proxy" Nov 02 15:47:27.406978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406694 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0be734-1c4a-4739-bfdb-821271658242" containerName="revoke-keys" Nov 02 15:47:27.406978 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.406707 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47059a3-a218-4d1c-9e27-5d4a5d968ecc" containerName="maas-api" Nov 02 15:47:27.410695 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.410671 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.464871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.464831 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms"] Nov 02 15:47:27.509414 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.509374 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:27.509673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.509470 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:47:27.542736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.542696 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.542929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.542761 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbl2\" (UniqueName: \"kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.542929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.542857 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.643913 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.643871 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbl2\" (UniqueName: \"kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.644130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.643926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.644130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.644013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.644647 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.644623 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.646469 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.646442 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.703235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.703157 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbl2\" (UniqueName: \"kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2\") pod \"maas-api-revoke-keys-e2e-worker-w1-624908-796ms\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.725061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:27.725015 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:27.935971 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:27.935922 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:47:28.104335 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:28.104236 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms"] Nov 02 15:47:28.110083 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:28.110054 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac8fc47_98b1_4a70_a545_77a6ffb35ae7.slice/crio-5c92bc4abda2cfafc05e4bca8329c9227b76fd881e728e8e816e34b51493d8b8 WatchSource:0}: Error finding container 5c92bc4abda2cfafc05e4bca8329c9227b76fd881e728e8e816e34b51493d8b8: Status 404 returned error can't find the container with id 5c92bc4abda2cfafc05e4bca8329c9227b76fd881e728e8e816e34b51493d8b8 Nov 02 15:47:28.275558 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:28.275501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" event={"ID":"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7","Type":"ContainerStarted","Data":"5c92bc4abda2cfafc05e4bca8329c9227b76fd881e728e8e816e34b51493d8b8"} Nov 02 15:47:28.814201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:28.814110 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:47:28.832761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:28.832732 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:47:29.285205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:29.285168 5091 generic.go:356] "Generic (PLEG): container finished" podID="4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" containerID="3ab0f87a15fb2d2baa3c7d9e554ebe376f2735791bdb7a2e2804ab413f1f8f02" exitCode=0 Nov 02 15:47:29.285430 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:29.285249 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" event={"ID":"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7","Type":"ContainerDied","Data":"3ab0f87a15fb2d2baa3c7d9e554ebe376f2735791bdb7a2e2804ab413f1f8f02"} Nov 02 15:47:30.178388 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.178352 5091 scope.go:122] "RemoveContainer" containerID="a5547735517a02d46fb47616a2acafd6200f9b7892967991dd7c75f530538b0b" Nov 02 15:47:30.188944 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.188918 5091 scope.go:122] "RemoveContainer" containerID="9769685228b47c0d7dfd2ad0d6b5f079a347c4c4f0a39f4a14f550716be90c42" Nov 02 15:47:30.431503 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.431432 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:30.575372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.575329 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca\") pod \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " Nov 02 15:47:30.575623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.575419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2\" (UniqueName: \"kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2\") pod \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " Nov 02 15:47:30.575623 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.575509 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca\") pod \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\" (UID: \"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7\") " Nov 02 15:47:30.576185 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.576153 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca" pod "4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" (UID: "4ac8fc47-98b1-4a70-a545-77a6ffb35ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:30.577737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.577701 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca" pod "4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" (UID: "4ac8fc47-98b1-4a70-a545-77a6ffb35ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:30.577876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.577845 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2" pod "4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" (UID: "4ac8fc47-98b1-4a70-a545-77a6ffb35ae7"). InnerVolumeSpecName "kube-api-access-8zbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:30.677412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.677377 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:30.677412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.677411 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:30.677412 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:30.677428 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbl2\" (UniqueName: \"kubernetes.io/projected/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7-kube-api-access-8zbl2\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:31.259620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:31.259583 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv"] Nov 02 15:47:31.299177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:31.299142 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" Nov 02 15:47:31.299177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:31.299168 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms" event={"ID":"4ac8fc47-98b1-4a70-a545-77a6ffb35ae7","Type":"ContainerDied","Data":"5c92bc4abda2cfafc05e4bca8329c9227b76fd881e728e8e816e34b51493d8b8"} Nov 02 15:47:31.299427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:31.299233 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c92bc4abda2cfafc05e4bca8329c9227b76fd881e728e8e816e34b51493d8b8" Nov 02 15:47:31.855826 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:31.855782 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn"] Nov 02 15:47:33.124361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:33.124320 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b"] Nov 02 15:47:33.124825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:33.124628 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" containerID="cri-o://c4835ad114ef6275f17cdf8821ab5f891396f5f16c6cec5d43f7a6713105c7e1" gracePeriod=30 Nov 02 15:47:34.159673 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.159624 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq"] Nov 02 15:47:34.160457 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.160425 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" containerName="revoke-keys" Nov 02 15:47:34.160691 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.160669 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" containerName="revoke-keys" Nov 02 15:47:34.160956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.160943 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" containerName="revoke-keys" Nov 02 15:47:34.164590 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.164564 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.214905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.214872 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:34.218610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.218462 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq"] Nov 02 15:47:34.262280 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.262237 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:47:34.329632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.329589 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.329833 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.329784 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7dnj\" (UniqueName: \"kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.329918 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.329897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.431386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.431293 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7dnj\" (UniqueName: \"kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.431386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.431366 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.431632 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.431407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.432230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.432197 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.433929 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.433901 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.492866 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.492825 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7dnj\" (UniqueName: \"kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj\") pod \"maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:34.778398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:34.778282 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:35.154507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:35.154467 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq"] Nov 02 15:47:35.160830 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:35.160797 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4d9739_e9a2_43c2_8731_42e9e7aff7a1.slice/crio-7576ed4c712b5b844683e1cab6ff7bead6a9f07f48c01fc87caadc3c2c6720d3 WatchSource:0}: Error finding container 7576ed4c712b5b844683e1cab6ff7bead6a9f07f48c01fc87caadc3c2c6720d3: Status 404 returned error can't find the container with id 7576ed4c712b5b844683e1cab6ff7bead6a9f07f48c01fc87caadc3c2c6720d3 Nov 02 15:47:35.326762 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:35.326721 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" event={"ID":"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1","Type":"ContainerStarted","Data":"7576ed4c712b5b844683e1cab6ff7bead6a9f07f48c01fc87caadc3c2c6720d3"} Nov 02 15:47:36.260096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:36.260036 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerName="payload-processing" containerID="cri-o://b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f" gracePeriod=25 Nov 02 15:47:36.334896 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:36.334859 5091 generic.go:356] "Generic (PLEG): container finished" podID="9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" containerID="e3e074428a9f957d1eefdbf463f476589b60248bad8b48b573473711f4b46c1a" exitCode=0 Nov 02 15:47:36.335089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:36.334947 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" event={"ID":"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1","Type":"ContainerDied","Data":"e3e074428a9f957d1eefdbf463f476589b60248bad8b48b573473711f4b46c1a"} Nov 02 15:47:37.479462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.479430 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:37.567663 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.567622 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca\") pod \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " Nov 02 15:47:37.567874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.567684 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca\") pod \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " Nov 02 15:47:37.567874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.567742 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj\" (UniqueName: \"kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj\") pod \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\" (UID: \"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1\") " Nov 02 15:47:37.568080 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.568054 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca" pod "9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" (UID: "9e4d9739-e9a2-43c2-8731-42e9e7aff7a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:37.568225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.568210 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:37.569895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.569870 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca" pod "9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" (UID: "9e4d9739-e9a2-43c2-8731-42e9e7aff7a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:37.570042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.569905 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj" pod "9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" (UID: "9e4d9739-e9a2-43c2-8731-42e9e7aff7a1"). InnerVolumeSpecName "kube-api-access-r7dnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:37.668878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.668832 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7dnj\" (UniqueName: \"kubernetes.io/projected/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-kube-api-access-r7dnj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:37.668878 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:37.668869 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.231941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.231853 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8"] Nov 02 15:47:38.333159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.333118 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d"] Nov 02 15:47:38.359709 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.359678 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" Nov 02 15:47:38.359883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.359672 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq" event={"ID":"9e4d9739-e9a2-43c2-8731-42e9e7aff7a1","Type":"ContainerDied","Data":"7576ed4c712b5b844683e1cab6ff7bead6a9f07f48c01fc87caadc3c2c6720d3"} Nov 02 15:47:38.359965 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.359903 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7576ed4c712b5b844683e1cab6ff7bead6a9f07f48c01fc87caadc3c2c6720d3" Nov 02 15:47:38.363009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.362930 5091 generic.go:356] "Generic (PLEG): container finished" podID="3763b422-89cf-4336-8097-cec4de9907bd" containerID="c4835ad114ef6275f17cdf8821ab5f891396f5f16c6cec5d43f7a6713105c7e1" exitCode=0 Nov 02 15:47:38.363009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.362988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" event={"ID":"3763b422-89cf-4336-8097-cec4de9907bd","Type":"ContainerDied","Data":"c4835ad114ef6275f17cdf8821ab5f891396f5f16c6cec5d43f7a6713105c7e1"} Nov 02 15:47:38.363265 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.363048 5091 scope.go:122] "RemoveContainer" containerID="95e0798e801764ce46fcfc6205278c2b03fe90b877d062d7a914dc1faaf01314" Nov 02 15:47:38.415060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.415032 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:47:38.493007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.492883 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493007 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.492960 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493029 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493060 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493139 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493232 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493270 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493299 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert\") pod \"3763b422-89cf-4336-8097-cec4de9907bd\" (UID: \"3763b422-89cf-4336-8097-cec4de9907bd\") " Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493391 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:38.493612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493585 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:38.494090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:38.494090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493801 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:38.494090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493825 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.494090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493848 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.494090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493861 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.494090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.493865 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:38.496569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.495589 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:38.496569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.495797 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:47:38.497090 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.497064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:38.497206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.497182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854" pod "3763b422-89cf-4336-8097-cec4de9907bd" (UID: "3763b422-89cf-4336-8097-cec4de9907bd"). InnerVolumeSpecName "kube-api-access-kd854". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:38.588505 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.588466 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb"] Nov 02 15:47:38.595461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.595330 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd854\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-kube-api-access-kd854\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.595461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.595372 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3763b422-89cf-4336-8097-cec4de9907bd-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.595461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.595392 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3763b422-89cf-4336-8097-cec4de9907bd-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.595461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.595405 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.595461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.595419 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3763b422-89cf-4336-8097-cec4de9907bd-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.595461 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.595432 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3763b422-89cf-4336-8097-cec4de9907bd-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:38.605151 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:38.605110 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6"] Nov 02 15:47:39.373598 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:39.373560 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" Nov 02 15:47:39.373827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:39.373570 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b" event={"ID":"3763b422-89cf-4336-8097-cec4de9907bd","Type":"ContainerDied","Data":"c52cdf50c3be4275d4e2f224b26be57e31aa0e5acf405923ac67155221b7ab34"} Nov 02 15:47:39.373827 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:39.373733 5091 scope.go:122] "RemoveContainer" containerID="c4835ad114ef6275f17cdf8821ab5f891396f5f16c6cec5d43f7a6713105c7e1" Nov 02 15:47:39.534777 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:39.534708 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b"] Nov 02 15:47:39.538845 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:39.538804 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b1ee37-openshift-default-5786db7d77-kxn5b"] Nov 02 15:47:39.962344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:39.962295 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3763b422-89cf-4336-8097-cec4de9907bd" path="/var/lib/kubelet/pods/3763b422-89cf-4336-8097-cec4de9907bd/volumes" Nov 02 15:47:40.976824 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.976784 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6"] Nov 02 15:47:40.977328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977311 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" Nov 02 15:47:40.977328 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977329 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" Nov 02 15:47:40.977454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977342 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" Nov 02 15:47:40.977454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977348 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" Nov 02 15:47:40.977454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977362 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" containerName="revoke-keys" Nov 02 15:47:40.977454 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977369 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" containerName="revoke-keys" Nov 02 15:47:40.977603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977473 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" Nov 02 15:47:40.977603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977483 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3763b422-89cf-4336-8097-cec4de9907bd" containerName="istio-proxy" Nov 02 15:47:40.977603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.977490 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" containerName="revoke-keys" Nov 02 15:47:40.982694 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:40.982661 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.024875 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.024836 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.025084 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.025054 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zv7\" (UniqueName: \"kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.025144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.025113 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.038248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.038213 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6"] Nov 02 15:47:41.081013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.080961 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:41.081215 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.081060 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:47:41.085230 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.085201 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs"] Nov 02 15:47:41.089277 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.089249 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.126481 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.126440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.126655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.126599 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zv7\" (UniqueName: \"kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.126655 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.126642 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.126803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.126691 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.126803 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.126726 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.126911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.126815 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c58gs\" (UniqueName: \"kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.127344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.127316 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.130135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.130104 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.133103 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.133072 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs"] Nov 02 15:47:41.187207 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.187168 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zv7\" (UniqueName: \"kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7\") pod \"maas-api-revoke-sub-e4396bf6-5c4t6\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.228405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.228293 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.228405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.228352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.228690 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.228423 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c58gs\" (UniqueName: \"kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.229165 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.229132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.230907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.230881 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.287998 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.287965 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c58gs\" (UniqueName: \"kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs\") pod \"maas-api-revoke-sub-296cfbd7-td5qs\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.299370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.299336 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:41.406883 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.405688 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:41.683475 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.683442 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6"] Nov 02 15:47:41.687555 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:41.687495 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e07e02_809e_48fd_a468_24cdaf8467b1.slice/crio-2f591d0fdab3b42c27f610e7de778b4ee75e6e5e0a0428031b2b771b12c51335 WatchSource:0}: Error finding container 2f591d0fdab3b42c27f610e7de778b4ee75e6e5e0a0428031b2b771b12c51335: Status 404 returned error can't find the container with id 2f591d0fdab3b42c27f610e7de778b4ee75e6e5e0a0428031b2b771b12c51335 Nov 02 15:47:41.795912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:41.795807 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs"] Nov 02 15:47:41.801075 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:41.801041 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf299dba4_aa6b_4dbb_b8eb_bf7475c94831.slice/crio-1ad16d6a563ce40adce71c90bc9111e2e480c3e35dac7ed789aa6887a71245c1 WatchSource:0}: Error finding container 1ad16d6a563ce40adce71c90bc9111e2e480c3e35dac7ed789aa6887a71245c1: Status 404 returned error can't find the container with id 1ad16d6a563ce40adce71c90bc9111e2e480c3e35dac7ed789aa6887a71245c1 Nov 02 15:47:42.398391 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:42.398280 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" event={"ID":"47e07e02-809e-48fd-a468-24cdaf8467b1","Type":"ContainerStarted","Data":"96b4d95c6c3e088911866e90abe31f37efd8e34b4e6149e02544bec1bcdb54af"} Nov 02 15:47:42.398844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:42.398389 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" event={"ID":"47e07e02-809e-48fd-a468-24cdaf8467b1","Type":"ContainerStarted","Data":"2f591d0fdab3b42c27f610e7de778b4ee75e6e5e0a0428031b2b771b12c51335"} Nov 02 15:47:42.400097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:42.400055 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" event={"ID":"f299dba4-aa6b-4dbb-b8eb-bf7475c94831","Type":"ContainerStarted","Data":"65de3d23533f9d8debc70d49e417087b6411c333f679792773f96294df468de0"} Nov 02 15:47:42.400269 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:42.400117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" event={"ID":"f299dba4-aa6b-4dbb-b8eb-bf7475c94831","Type":"ContainerStarted","Data":"1ad16d6a563ce40adce71c90bc9111e2e480c3e35dac7ed789aa6887a71245c1"} Nov 02 15:47:42.510393 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:42.510325 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" podStartSLOduration=2.5103049889999998 podStartE2EDuration="2.510304989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:42.508122154 +0000 UTC m=+3016.672674696" watchObservedRunningTime="2026-11-02 15:47:42.510304989 +0000 UTC m=+3016.674857529" Nov 02 15:47:42.621227 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:42.621159 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" podStartSLOduration=1.621141112 podStartE2EDuration="1.621141112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:42.619429354 +0000 UTC m=+3016.783981895" watchObservedRunningTime="2026-11-02 15:47:42.621141112 +0000 UTC m=+3016.785693652" Nov 02 15:47:43.232396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.232324 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" podUID="bf255e40-df8e-45d8-9eae-9c9bdcb78a29" containerName="payload-processing" containerID="cri-o://32de910863a57acbea0200bfa29555cd5969faa308ce41218872fa9a404fece9" gracePeriod=25 Nov 02 15:47:43.333480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.333409 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerName="payload-pre-processing" containerID="cri-o://6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4" gracePeriod=25 Nov 02 15:47:43.408990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.408951 5091 generic.go:356] "Generic (PLEG): container finished" podID="47e07e02-809e-48fd-a468-24cdaf8467b1" containerID="96b4d95c6c3e088911866e90abe31f37efd8e34b4e6149e02544bec1bcdb54af" exitCode=0 Nov 02 15:47:43.409465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.409034 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" event={"ID":"47e07e02-809e-48fd-a468-24cdaf8467b1","Type":"ContainerDied","Data":"96b4d95c6c3e088911866e90abe31f37efd8e34b4e6149e02544bec1bcdb54af"} Nov 02 15:47:43.411118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.411085 5091 generic.go:356] "Generic (PLEG): container finished" podID="f299dba4-aa6b-4dbb-b8eb-bf7475c94831" containerID="65de3d23533f9d8debc70d49e417087b6411c333f679792773f96294df468de0" exitCode=0 Nov 02 15:47:43.411261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.411169 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" event={"ID":"f299dba4-aa6b-4dbb-b8eb-bf7475c94831","Type":"ContainerDied","Data":"65de3d23533f9d8debc70d49e417087b6411c333f679792773f96294df468de0"} Nov 02 15:47:43.413612 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.413585 5091 generic.go:356] "Generic (PLEG): container finished" podID="bf255e40-df8e-45d8-9eae-9c9bdcb78a29" containerID="32de910863a57acbea0200bfa29555cd5969faa308ce41218872fa9a404fece9" exitCode=0 Nov 02 15:47:43.413722 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.413667 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" event={"ID":"bf255e40-df8e-45d8-9eae-9c9bdcb78a29","Type":"ContainerDied","Data":"32de910863a57acbea0200bfa29555cd5969faa308ce41218872fa9a404fece9"} Nov 02 15:47:43.520315 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.520245 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:47:43.559706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.559672 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume\") pod \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " Nov 02 15:47:43.559924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.559816 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82\" (UniqueName: \"kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82\") pod \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\" (UID: \"bf255e40-df8e-45d8-9eae-9c9bdcb78a29\") " Nov 02 15:47:43.560127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.560102 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume" pod "bf255e40-df8e-45d8-9eae-9c9bdcb78a29" (UID: "bf255e40-df8e-45d8-9eae-9c9bdcb78a29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:43.561951 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.561927 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82" pod "bf255e40-df8e-45d8-9eae-9c9bdcb78a29" (UID: "bf255e40-df8e-45d8-9eae-9c9bdcb78a29"). InnerVolumeSpecName "kube-api-access-7kz82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:43.661398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.661351 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kz82\" (UniqueName: \"kubernetes.io/projected/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-kube-api-access-7kz82\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:43.661398 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:43.661394 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf255e40-df8e-45d8-9eae-9c9bdcb78a29-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:44.421588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.421546 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" event={"ID":"bf255e40-df8e-45d8-9eae-9c9bdcb78a29","Type":"ContainerDied","Data":"1795478c70533345187ff8d893bb43dbe10f42123e5a42fb9c39240f49b8eeee"} Nov 02 15:47:44.421588 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.421587 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8" Nov 02 15:47:44.422157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.421638 5091 scope.go:122] "RemoveContainer" containerID="32de910863a57acbea0200bfa29555cd5969faa308ce41218872fa9a404fece9" Nov 02 15:47:44.499973 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.499928 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:44.582746 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.582691 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8"] Nov 02 15:47:44.587323 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.587286 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-624908-8595ff8b5f-lppt8"] Nov 02 15:47:44.590076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.590041 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:44.632773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.632744 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:44.674055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.673947 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7\" (UniqueName: \"kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7\") pod \"47e07e02-809e-48fd-a468-24cdaf8467b1\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " Nov 02 15:47:44.674055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.674043 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca\") pod \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " Nov 02 15:47:44.674319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.674089 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs\" (UniqueName: \"kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs\") pod \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " Nov 02 15:47:44.674319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.674127 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca\") pod \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\" (UID: \"f299dba4-aa6b-4dbb-b8eb-bf7475c94831\") " Nov 02 15:47:44.674319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.674236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca\") pod \"47e07e02-809e-48fd-a468-24cdaf8467b1\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " Nov 02 15:47:44.674319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.674301 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca\") pod \"47e07e02-809e-48fd-a468-24cdaf8467b1\" (UID: \"47e07e02-809e-48fd-a468-24cdaf8467b1\") " Nov 02 15:47:44.674895 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.674822 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca" pod "47e07e02-809e-48fd-a468-24cdaf8467b1" (UID: "47e07e02-809e-48fd-a468-24cdaf8467b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:44.675270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.675189 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca" pod "f299dba4-aa6b-4dbb-b8eb-bf7475c94831" (UID: "f299dba4-aa6b-4dbb-b8eb-bf7475c94831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:44.677073 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.677015 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca" pod "47e07e02-809e-48fd-a468-24cdaf8467b1" (UID: "47e07e02-809e-48fd-a468-24cdaf8467b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:44.677073 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.677046 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca" pod "f299dba4-aa6b-4dbb-b8eb-bf7475c94831" (UID: "f299dba4-aa6b-4dbb-b8eb-bf7475c94831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:44.677205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.677063 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7" pod "47e07e02-809e-48fd-a468-24cdaf8467b1" (UID: "47e07e02-809e-48fd-a468-24cdaf8467b1"). InnerVolumeSpecName "kube-api-access-c2zv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:44.677392 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.677366 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs" pod "f299dba4-aa6b-4dbb-b8eb-bf7475c94831" (UID: "f299dba4-aa6b-4dbb-b8eb-bf7475c94831"). InnerVolumeSpecName "kube-api-access-c58gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:44.775578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.775509 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:44.775578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.775572 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zv7\" (UniqueName: \"kubernetes.io/projected/47e07e02-809e-48fd-a468-24cdaf8467b1-kube-api-access-c2zv7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:44.775578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.775583 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:44.775828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.775593 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c58gs\" (UniqueName: \"kubernetes.io/projected/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-kube-api-access-c58gs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:44.775828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.775604 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f299dba4-aa6b-4dbb-b8eb-bf7475c94831-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:44.775828 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:44.775613 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47e07e02-809e-48fd-a468-24cdaf8467b1-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:45.430569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.430534 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" Nov 02 15:47:45.430569 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.430549 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6" event={"ID":"47e07e02-809e-48fd-a468-24cdaf8467b1","Type":"ContainerDied","Data":"2f591d0fdab3b42c27f610e7de778b4ee75e6e5e0a0428031b2b771b12c51335"} Nov 02 15:47:45.431093 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.430610 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f591d0fdab3b42c27f610e7de778b4ee75e6e5e0a0428031b2b771b12c51335" Nov 02 15:47:45.432389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.432363 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" Nov 02 15:47:45.432389 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.432374 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs" event={"ID":"f299dba4-aa6b-4dbb-b8eb-bf7475c94831","Type":"ContainerDied","Data":"1ad16d6a563ce40adce71c90bc9111e2e480c3e35dac7ed789aa6887a71245c1"} Nov 02 15:47:45.432638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.432432 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad16d6a563ce40adce71c90bc9111e2e480c3e35dac7ed789aa6887a71245c1" Nov 02 15:47:45.970187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:45.970148 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf255e40-df8e-45d8-9eae-9c9bdcb78a29" path="/var/lib/kubelet/pods/bf255e40-df8e-45d8-9eae-9c9bdcb78a29/volumes" Nov 02 15:47:46.847032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.846988 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw"] Nov 02 15:47:46.847571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847508 5091 cpu_manager.go:414] "containerMap: removing container" podUID="47e07e02-809e-48fd-a468-24cdaf8467b1" containerName="revoke-keys" Nov 02 15:47:46.847571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847537 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47e07e02-809e-48fd-a468-24cdaf8467b1" containerName="revoke-keys" Nov 02 15:47:46.847571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847565 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f299dba4-aa6b-4dbb-b8eb-bf7475c94831" containerName="revoke-keys" Nov 02 15:47:46.847571 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847572 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f299dba4-aa6b-4dbb-b8eb-bf7475c94831" containerName="revoke-keys" Nov 02 15:47:46.847782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847591 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bf255e40-df8e-45d8-9eae-9c9bdcb78a29" containerName="payload-processing" Nov 02 15:47:46.847782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847597 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf255e40-df8e-45d8-9eae-9c9bdcb78a29" containerName="payload-processing" Nov 02 15:47:46.847782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847681 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="47e07e02-809e-48fd-a468-24cdaf8467b1" containerName="revoke-keys" Nov 02 15:47:46.847782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847692 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f299dba4-aa6b-4dbb-b8eb-bf7475c94831" containerName="revoke-keys" Nov 02 15:47:46.847782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.847702 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf255e40-df8e-45d8-9eae-9c9bdcb78a29" containerName="payload-processing" Nov 02 15:47:46.852253 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.852232 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.885846 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.885804 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" containerID="cri-o://426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed" gracePeriod=45 Nov 02 15:47:46.896652 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.896614 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.896818 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.896792 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89cxn\" (UniqueName: \"kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.896879 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.896829 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.899056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.899032 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2"] Nov 02 15:47:46.903215 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.903076 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:46.905891 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.905733 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw"] Nov 02 15:47:46.943641 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.943607 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:47:46.943848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.943666 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:47:46.964909 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.964851 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2"] Nov 02 15:47:46.998351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.998306 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjj52\" (UniqueName: \"kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:46.998547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.998426 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:46.998547 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.998500 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:46.998713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.998686 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89cxn\" (UniqueName: \"kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.998790 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.998732 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.998848 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.998811 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:46.999713 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:46.999686 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:47.001286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.001261 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:47.058429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.058397 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89cxn\" (UniqueName: \"kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn\") pod \"maas-api-revoke-sub-5820afe5-zxjrw\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:47.100127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.100025 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.100284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.100200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjj52\" (UniqueName: \"kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.100336 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.100290 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.101198 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.101164 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.102819 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.102784 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.167239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.167107 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:47.170685 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.170653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjj52\" (UniqueName: \"kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52\") pod \"maas-api-revoke-sub-11fa493e-jxhf2\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.180801 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.180233 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:47:47.269622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.269576 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:47.302707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.302668 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs\") pod \"f977fedd-4641-483d-8c88-1fd88475c4b4\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " Nov 02 15:47:47.302915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.302729 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home\") pod \"f977fedd-4641-483d-8c88-1fd88475c4b4\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " Nov 02 15:47:47.302915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.302814 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl\" (UniqueName: \"kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl\") pod \"f977fedd-4641-483d-8c88-1fd88475c4b4\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " Nov 02 15:47:47.302915 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.302852 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm\") pod \"f977fedd-4641-483d-8c88-1fd88475c4b4\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " Nov 02 15:47:47.303095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.302957 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir\") pod \"f977fedd-4641-483d-8c88-1fd88475c4b4\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " Nov 02 15:47:47.303095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.303030 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache\") pod \"f977fedd-4641-483d-8c88-1fd88475c4b4\" (UID: \"f977fedd-4641-483d-8c88-1fd88475c4b4\") " Nov 02 15:47:47.303219 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.303192 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home" pod "f977fedd-4641-483d-8c88-1fd88475c4b4" (UID: "f977fedd-4641-483d-8c88-1fd88475c4b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.303567 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.303503 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.303890 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.303851 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache" pod "f977fedd-4641-483d-8c88-1fd88475c4b4" (UID: "f977fedd-4641-483d-8c88-1fd88475c4b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.307589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.307545 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir" pod "f977fedd-4641-483d-8c88-1fd88475c4b4" (UID: "f977fedd-4641-483d-8c88-1fd88475c4b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.307835 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.307804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl" pod "f977fedd-4641-483d-8c88-1fd88475c4b4" (UID: "f977fedd-4641-483d-8c88-1fd88475c4b4"). InnerVolumeSpecName "kube-api-access-xztrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:47.308233 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.308208 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm" pod "f977fedd-4641-483d-8c88-1fd88475c4b4" (UID: "f977fedd-4641-483d-8c88-1fd88475c4b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:47.310089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.310063 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs" pod "f977fedd-4641-483d-8c88-1fd88475c4b4" (UID: "f977fedd-4641-483d-8c88-1fd88475c4b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:47.404342 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.404300 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xztrl\" (UniqueName: \"kubernetes.io/projected/f977fedd-4641-483d-8c88-1fd88475c4b4-kube-api-access-xztrl\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.404342 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.404339 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.404592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.404353 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.404592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.404366 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f977fedd-4641-483d-8c88-1fd88475c4b4-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.404592 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.404378 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f977fedd-4641-483d-8c88-1fd88475c4b4-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:47.453992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.453897 5091 generic.go:356] "Generic (PLEG): container finished" podID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerID="426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed" exitCode=0 Nov 02 15:47:47.454350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.454315 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" event={"ID":"f977fedd-4641-483d-8c88-1fd88475c4b4","Type":"ContainerDied","Data":"426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed"} Nov 02 15:47:47.454459 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.454376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" event={"ID":"f977fedd-4641-483d-8c88-1fd88475c4b4","Type":"ContainerDied","Data":"32f77039790759550d756200cd0515ea9609d3a4e073ed80286daa6cc473b3b2"} Nov 02 15:47:47.454459 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.454401 5091 scope.go:122] "RemoveContainer" containerID="426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed" Nov 02 15:47:47.454635 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.454582 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn" Nov 02 15:47:47.486773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.486745 5091 scope.go:122] "RemoveContainer" containerID="426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed" Nov 02 15:47:47.487173 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:47.487144 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed\": container with ID starting with 426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed not found: ID does not exist" containerID="426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed" Nov 02 15:47:47.487297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.487185 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed"} err="failed to get container status \"426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed\": rpc error: code = NotFound desc = could not find container \"426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed\": container with ID starting with 426b1cfbf32175bb240827c925a1e7922724dbb8b3414d45bb08f1decf3675ed not found: ID does not exist" Nov 02 15:47:47.560809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.560766 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw"] Nov 02 15:47:47.567730 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:47.567699 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8216f8b_57f7_4332_9092_e7fef23819d2.slice/crio-c055d80a15721b25a04bd50569b743486f8d11df7f31e58cc85707b7292e3ee8 WatchSource:0}: Error finding container c055d80a15721b25a04bd50569b743486f8d11df7f31e58cc85707b7292e3ee8: Status 404 returned error can't find the container with id c055d80a15721b25a04bd50569b743486f8d11df7f31e58cc85707b7292e3ee8 Nov 02 15:47:47.620193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.620153 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn"] Nov 02 15:47:47.627282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.627241 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-624908/distinct-2-w1-624908-kserve-77b5d689b4-mmwbn"] Nov 02 15:47:47.658361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.658321 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2"] Nov 02 15:47:47.661265 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:47:47.661236 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaddab77_172e_4062_b78b_d116ea0b01b5.slice/crio-67948df81456b8d96814e51369c3988fff43a645c5f02de453d2fcfaae96f41a WatchSource:0}: Error finding container 67948df81456b8d96814e51369c3988fff43a645c5f02de453d2fcfaae96f41a: Status 404 returned error can't find the container with id 67948df81456b8d96814e51369c3988fff43a645c5f02de453d2fcfaae96f41a Nov 02 15:47:47.960241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:47.960194 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" path="/var/lib/kubelet/pods/f977fedd-4641-483d-8c88-1fd88475c4b4/volumes" Nov 02 15:47:48.464983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:48.464938 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" event={"ID":"e8216f8b-57f7-4332-9092-e7fef23819d2","Type":"ContainerStarted","Data":"e814ed26f8d63c544040bcbdf54727f39e361eb58a90a9340757444bdfffccfb"} Nov 02 15:47:48.465294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:48.465269 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" event={"ID":"e8216f8b-57f7-4332-9092-e7fef23819d2","Type":"ContainerStarted","Data":"c055d80a15721b25a04bd50569b743486f8d11df7f31e58cc85707b7292e3ee8"} Nov 02 15:47:48.467002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:48.466961 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" event={"ID":"eaddab77-172e-4062-b78b-d116ea0b01b5","Type":"ContainerStarted","Data":"5b5dfe408e2d2ae33bf500a094934fb01aeef4cc7ab87b3e43f5dfb1b37962e5"} Nov 02 15:47:48.467163 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:48.467021 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" event={"ID":"eaddab77-172e-4062-b78b-d116ea0b01b5","Type":"ContainerStarted","Data":"67948df81456b8d96814e51369c3988fff43a645c5f02de453d2fcfaae96f41a"} Nov 02 15:47:48.578097 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:48.578036 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" podStartSLOduration=2.578022367 podStartE2EDuration="2.578022367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:48.576165408 +0000 UTC m=+3022.740717948" watchObservedRunningTime="2026-11-02 15:47:48.578022367 +0000 UTC m=+3022.742574930" Nov 02 15:47:48.688872 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:48.688804 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" podStartSLOduration=2.688782481 podStartE2EDuration="2.688782481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:47:48.686635217 +0000 UTC m=+3022.851187772" watchObservedRunningTime="2026-11-02 15:47:48.688782481 +0000 UTC m=+3022.853335024" Nov 02 15:47:49.477875 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:49.477833 5091 generic.go:356] "Generic (PLEG): container finished" podID="e8216f8b-57f7-4332-9092-e7fef23819d2" containerID="e814ed26f8d63c544040bcbdf54727f39e361eb58a90a9340757444bdfffccfb" exitCode=0 Nov 02 15:47:49.480544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:49.477913 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" event={"ID":"e8216f8b-57f7-4332-9092-e7fef23819d2","Type":"ContainerDied","Data":"e814ed26f8d63c544040bcbdf54727f39e361eb58a90a9340757444bdfffccfb"} Nov 02 15:47:49.480544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:49.479656 5091 generic.go:356] "Generic (PLEG): container finished" podID="eaddab77-172e-4062-b78b-d116ea0b01b5" containerID="5b5dfe408e2d2ae33bf500a094934fb01aeef4cc7ab87b3e43f5dfb1b37962e5" exitCode=0 Nov 02 15:47:49.480544 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:49.479736 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" event={"ID":"eaddab77-172e-4062-b78b-d116ea0b01b5","Type":"ContainerDied","Data":"5b5dfe408e2d2ae33bf500a094934fb01aeef4cc7ab87b3e43f5dfb1b37962e5"} Nov 02 15:47:50.655441 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.655406 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:50.698908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.698878 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:50.738651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.738539 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn\" (UniqueName: \"kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn\") pod \"e8216f8b-57f7-4332-9092-e7fef23819d2\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " Nov 02 15:47:50.738651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.738629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca\") pod \"e8216f8b-57f7-4332-9092-e7fef23819d2\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " Nov 02 15:47:50.738928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.738678 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52\" (UniqueName: \"kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52\") pod \"eaddab77-172e-4062-b78b-d116ea0b01b5\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " Nov 02 15:47:50.738928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.738721 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca\") pod \"eaddab77-172e-4062-b78b-d116ea0b01b5\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " Nov 02 15:47:50.738928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.738781 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca\") pod \"eaddab77-172e-4062-b78b-d116ea0b01b5\" (UID: \"eaddab77-172e-4062-b78b-d116ea0b01b5\") " Nov 02 15:47:50.738928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.738817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca\") pod \"e8216f8b-57f7-4332-9092-e7fef23819d2\" (UID: \"e8216f8b-57f7-4332-9092-e7fef23819d2\") " Nov 02 15:47:50.739748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.739717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca" pod "e8216f8b-57f7-4332-9092-e7fef23819d2" (UID: "e8216f8b-57f7-4332-9092-e7fef23819d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:50.741748 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.741693 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca" pod "eaddab77-172e-4062-b78b-d116ea0b01b5" (UID: "eaddab77-172e-4062-b78b-d116ea0b01b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:50.757340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.750553 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52" pod "eaddab77-172e-4062-b78b-d116ea0b01b5" (UID: "eaddab77-172e-4062-b78b-d116ea0b01b5"). InnerVolumeSpecName "kube-api-access-gjj52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:50.757340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.750705 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn" pod "e8216f8b-57f7-4332-9092-e7fef23819d2" (UID: "e8216f8b-57f7-4332-9092-e7fef23819d2"). InnerVolumeSpecName "kube-api-access-89cxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:50.757340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.751659 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca" pod "e8216f8b-57f7-4332-9092-e7fef23819d2" (UID: "e8216f8b-57f7-4332-9092-e7fef23819d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:50.757340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.752748 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca" pod "eaddab77-172e-4062-b78b-d116ea0b01b5" (UID: "eaddab77-172e-4062-b78b-d116ea0b01b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:50.840749 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.840710 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89cxn\" (UniqueName: \"kubernetes.io/projected/e8216f8b-57f7-4332-9092-e7fef23819d2-kube-api-access-89cxn\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:50.840749 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.840746 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:50.840749 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.840757 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjj52\" (UniqueName: \"kubernetes.io/projected/eaddab77-172e-4062-b78b-d116ea0b01b5-kube-api-access-gjj52\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:50.841015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.840767 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:50.841015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.840776 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaddab77-172e-4062-b78b-d116ea0b01b5-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:50.841015 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:50.840786 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8216f8b-57f7-4332-9092-e7fef23819d2-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:51.168384 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.168338 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:51.494181 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.494094 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" Nov 02 15:47:51.494358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.494098 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw" event={"ID":"e8216f8b-57f7-4332-9092-e7fef23819d2","Type":"ContainerDied","Data":"c055d80a15721b25a04bd50569b743486f8d11df7f31e58cc85707b7292e3ee8"} Nov 02 15:47:51.494358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.494223 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c055d80a15721b25a04bd50569b743486f8d11df7f31e58cc85707b7292e3ee8" Nov 02 15:47:51.495620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.495592 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" Nov 02 15:47:51.495620 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.495594 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2" event={"ID":"eaddab77-172e-4062-b78b-d116ea0b01b5","Type":"ContainerDied","Data":"67948df81456b8d96814e51369c3988fff43a645c5f02de453d2fcfaae96f41a"} Nov 02 15:47:51.495809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:51.495645 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67948df81456b8d96814e51369c3988fff43a645c5f02de453d2fcfaae96f41a" Nov 02 15:47:53.612721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.612670 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" containerID="cri-o://cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770" gracePeriod=45 Nov 02 15:47:53.641304 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.641262 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" containerID="cri-o://f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263" gracePeriod=45 Nov 02 15:47:53.643022 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:53.642986 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2d82f4_e05a_4737_970c_38a126c560b8.slice/crio-cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770.scope\": RecentStats: unable to find data in memory cache]" Nov 02 15:47:53.670622 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.670568 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused" Nov 02 15:47:53.911241 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.911205 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:47:53.940698 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.940674 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:47:53.974789 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.974754 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7\" (UniqueName: \"kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7\") pod \"0a2d82f4-e05a-4737-970c-38a126c560b8\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " Nov 02 15:47:53.974956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.974811 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm\") pod \"0a2d82f4-e05a-4737-970c-38a126c560b8\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " Nov 02 15:47:53.974956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.974873 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache\") pod \"0a2d82f4-e05a-4737-970c-38a126c560b8\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " Nov 02 15:47:53.974956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.974913 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home\") pod \"0a2d82f4-e05a-4737-970c-38a126c560b8\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " Nov 02 15:47:53.974956 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.974944 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir\") pod \"0a2d82f4-e05a-4737-970c-38a126c560b8\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " Nov 02 15:47:53.975128 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975019 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs\") pod \"0a2d82f4-e05a-4737-970c-38a126c560b8\" (UID: \"0a2d82f4-e05a-4737-970c-38a126c560b8\") " Nov 02 15:47:53.975225 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975202 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache" pod "0a2d82f4-e05a-4737-970c-38a126c560b8" (UID: "0a2d82f4-e05a-4737-970c-38a126c560b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:53.975300 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975230 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home" pod "0a2d82f4-e05a-4737-970c-38a126c560b8" (UID: "0a2d82f4-e05a-4737-970c-38a126c560b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:53.975354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975310 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir" pod "0a2d82f4-e05a-4737-970c-38a126c560b8" (UID: "0a2d82f4-e05a-4737-970c-38a126c560b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:53.975499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975477 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:53.975499 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975499 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:53.975675 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.975542 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:53.977217 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.977184 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs" pod "0a2d82f4-e05a-4737-970c-38a126c560b8" (UID: "0a2d82f4-e05a-4737-970c-38a126c560b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:53.977313 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.977218 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm" pod "0a2d82f4-e05a-4737-970c-38a126c560b8" (UID: "0a2d82f4-e05a-4737-970c-38a126c560b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:53.977313 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:53.977242 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7" pod "0a2d82f4-e05a-4737-970c-38a126c560b8" (UID: "0a2d82f4-e05a-4737-970c-38a126c560b8"). InnerVolumeSpecName "kube-api-access-bfqp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:54.076865 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.076827 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home\") pod \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " Nov 02 15:47:54.077053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.076945 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs\") pod \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " Nov 02 15:47:54.077053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.076993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir\") pod \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " Nov 02 15:47:54.077053 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077028 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache\") pod \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " Nov 02 15:47:54.077270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077063 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm\") pod \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " Nov 02 15:47:54.077270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077098 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj\" (UniqueName: \"kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj\") pod \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\" (UID: \"bf795812-c210-47d8-8a8e-d4aa5546a4a8\") " Nov 02 15:47:54.078020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077630 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2d82f4-e05a-4737-970c-38a126c560b8-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.078020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077654 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2d82f4-e05a-4737-970c-38a126c560b8-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.078020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077667 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfqp7\" (UniqueName: \"kubernetes.io/projected/0a2d82f4-e05a-4737-970c-38a126c560b8-kube-api-access-bfqp7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.078020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077692 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir" pod "bf795812-c210-47d8-8a8e-d4aa5546a4a8" (UID: "bf795812-c210-47d8-8a8e-d4aa5546a4a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.078020 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.077964 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache" pod "bf795812-c210-47d8-8a8e-d4aa5546a4a8" (UID: "bf795812-c210-47d8-8a8e-d4aa5546a4a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.078545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.078155 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home" pod "bf795812-c210-47d8-8a8e-d4aa5546a4a8" (UID: "bf795812-c210-47d8-8a8e-d4aa5546a4a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.080095 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.080071 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm" pod "bf795812-c210-47d8-8a8e-d4aa5546a4a8" (UID: "bf795812-c210-47d8-8a8e-d4aa5546a4a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:47:54.081651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.081498 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs" pod "bf795812-c210-47d8-8a8e-d4aa5546a4a8" (UID: "bf795812-c210-47d8-8a8e-d4aa5546a4a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:47:54.081651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.081609 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj" pod "bf795812-c210-47d8-8a8e-d4aa5546a4a8" (UID: "bf795812-c210-47d8-8a8e-d4aa5546a4a8"). InnerVolumeSpecName "kube-api-access-cttnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:54.179082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.178993 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.179082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.179039 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.179082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.179053 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.179082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.179067 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.179082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.179083 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf795812-c210-47d8-8a8e-d4aa5546a4a8-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.179377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.179098 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttnj\" (UniqueName: \"kubernetes.io/projected/bf795812-c210-47d8-8a8e-d4aa5546a4a8-kube-api-access-cttnj\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:54.455092 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.455013 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2"] Nov 02 15:47:54.457498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.457475 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f6b245-kjzx2"] Nov 02 15:47:54.499032 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.498995 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:47:54.514621 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.514592 5091 generic.go:356] "Generic (PLEG): container finished" podID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerID="f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263" exitCode=0 Nov 02 15:47:54.514778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.514642 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" Nov 02 15:47:54.514778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.514678 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" event={"ID":"bf795812-c210-47d8-8a8e-d4aa5546a4a8","Type":"ContainerDied","Data":"f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263"} Nov 02 15:47:54.514778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.514722 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6" event={"ID":"bf795812-c210-47d8-8a8e-d4aa5546a4a8","Type":"ContainerDied","Data":"fc991b34a0b51b23991966160abdfe665a36108f82102b2cd67b2a3c0e9ec8bb"} Nov 02 15:47:54.514778 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.514744 5091 scope.go:122] "RemoveContainer" containerID="f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263" Nov 02 15:47:54.516155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.516127 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerID="cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770" exitCode=0 Nov 02 15:47:54.516275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.516204 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" Nov 02 15:47:54.516275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.516245 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" event={"ID":"0a2d82f4-e05a-4737-970c-38a126c560b8","Type":"ContainerDied","Data":"cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770"} Nov 02 15:47:54.516370 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.516287 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb" event={"ID":"0a2d82f4-e05a-4737-970c-38a126c560b8","Type":"ContainerDied","Data":"5359c501c13732d499827aa1a03687488e97fe2e91e98b9abdea45ca3bf86b7b"} Nov 02 15:47:54.525118 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.525094 5091 scope.go:122] "RemoveContainer" containerID="f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263" Nov 02 15:47:54.525490 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:54.525387 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263\": container with ID starting with f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263 not found: ID does not exist" containerID="f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263" Nov 02 15:47:54.525490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.525420 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263"} err="failed to get container status \"f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263\": rpc error: code = NotFound desc = could not find container \"f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263\": container with ID starting with f0308a02d0fe31ba65142bb0607aaef7525e00ac320b8e739e01f1bb0d6e4263 not found: ID does not exist" Nov 02 15:47:54.525490 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.525445 5091 scope.go:122] "RemoveContainer" containerID="cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770" Nov 02 15:47:54.534270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.534250 5091 scope.go:122] "RemoveContainer" containerID="cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770" Nov 02 15:47:54.534540 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:54.534509 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770\": container with ID starting with cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770 not found: ID does not exist" containerID="cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770" Nov 02 15:47:54.534602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.534548 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770"} err="failed to get container status \"cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770\": rpc error: code = NotFound desc = could not find container \"cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770\": container with ID starting with cce3d900e33295b9bdfaca975295efdd279829c6c0c770d902db100a5f3f3770 not found: ID does not exist" Nov 02 15:47:54.674560 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.674495 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6"] Nov 02 15:47:54.677262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.677240 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/simulator-w4-8a2e6f-kserve-64b895b68f-cfmq6"] Nov 02 15:47:54.780224 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.780093 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb"] Nov 02 15:47:54.784489 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:54.784453 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8a2e6f/distinct-w4-8a2e6f-kserve-67dfb68b5f-xcnrb"] Nov 02 15:47:55.959507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:55.959474 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" path="/var/lib/kubelet/pods/0a2d82f4-e05a-4737-970c-38a126c560b8/volumes" Nov 02 15:47:55.959971 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:55.959953 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0be734-1c4a-4739-bfdb-821271658242" path="/var/lib/kubelet/pods/bb0be734-1c4a-4739-bfdb-821271658242/volumes" Nov 02 15:47:55.960330 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:55.960317 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" path="/var/lib/kubelet/pods/bf795812-c210-47d8-8a8e-d4aa5546a4a8/volumes" Nov 02 15:47:56.532276 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.532188 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:47:56.532942 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.532914 5091 generic.go:356] "Generic (PLEG): container finished" podID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerID="b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f" exitCode=0 Nov 02 15:47:56.533065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.532984 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" event={"ID":"a27b3e27-f31e-40e8-991f-814e9922e08e","Type":"ContainerDied","Data":"b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f"} Nov 02 15:47:56.533065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.533040 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" event={"ID":"a27b3e27-f31e-40e8-991f-814e9922e08e","Type":"ContainerDied","Data":"c4d3bf090df03a9232e5c78cf0c07362b6a3c0615bf6a7856735f74f9f1d3a2b"} Nov 02 15:47:56.533065 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.533063 5091 scope.go:122] "RemoveContainer" containerID="b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f" Nov 02 15:47:56.543589 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.543560 5091 scope.go:122] "RemoveContainer" containerID="b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f" Nov 02 15:47:56.543974 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:56.543952 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f\": container with ID starting with b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f not found: ID does not exist" containerID="b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f" Nov 02 15:47:56.544052 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.543989 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f"} err="failed to get container status \"b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f\": rpc error: code = NotFound desc = could not find container \"b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f\": container with ID starting with b3119d8028b9c38c87e88b1419c36196dc17338301c4f4ab732b81b9708db99f not found: ID does not exist" Nov 02 15:47:56.602898 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.602860 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume\") pod \"a27b3e27-f31e-40e8-991f-814e9922e08e\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " Nov 02 15:47:56.603096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.602956 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q\" (UniqueName: \"kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q\") pod \"a27b3e27-f31e-40e8-991f-814e9922e08e\" (UID: \"a27b3e27-f31e-40e8-991f-814e9922e08e\") " Nov 02 15:47:56.603290 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.603267 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume" pod "a27b3e27-f31e-40e8-991f-814e9922e08e" (UID: "a27b3e27-f31e-40e8-991f-814e9922e08e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:47:56.605137 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.605112 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q" pod "a27b3e27-f31e-40e8-991f-814e9922e08e" (UID: "a27b3e27-f31e-40e8-991f-814e9922e08e"). InnerVolumeSpecName "kube-api-access-4pj4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:47:56.704630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.704586 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pj4q\" (UniqueName: \"kubernetes.io/projected/a27b3e27-f31e-40e8-991f-814e9922e08e-kube-api-access-4pj4q\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:56.704630 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:56.704625 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27b3e27-f31e-40e8-991f-814e9922e08e-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:47:57.540794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:57.540757 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv" Nov 02 15:47:57.700921 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:57.700876 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv"] Nov 02 15:47:57.746534 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:57.746475 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f6b245-65994fd897-4tgdv"] Nov 02 15:47:57.939408 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:47:57.939368 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:47:57.960737 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:47:57.960698 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" path="/var/lib/kubelet/pods/a27b3e27-f31e-40e8-991f-814e9922e08e/volumes" Nov 02 15:48:01.167784 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:01.167742 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 15:48:02.905653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.905621 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s"] Nov 02 15:48:02.906077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906064 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906078 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906092 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906098 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906104 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e8216f8b-57f7-4332-9092-e7fef23819d2" containerName="revoke-keys" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906108 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8216f8b-57f7-4332-9092-e7fef23819d2" containerName="revoke-keys" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906118 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" Nov 02 15:48:02.906124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906123 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906145 5091 cpu_manager.go:414] "containerMap: removing container" podUID="eaddab77-172e-4062-b78b-d116ea0b01b5" containerName="revoke-keys" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906150 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaddab77-172e-4062-b78b-d116ea0b01b5" containerName="revoke-keys" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906159 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerName="payload-processing" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906164 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerName="payload-processing" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906247 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaddab77-172e-4062-b78b-d116ea0b01b5" containerName="revoke-keys" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906256 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27b3e27-f31e-40e8-991f-814e9922e08e" containerName="payload-processing" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906267 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf795812-c210-47d8-8a8e-d4aa5546a4a8" containerName="main" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906273 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2d82f4-e05a-4737-970c-38a126c560b8" containerName="main" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906278 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8216f8b-57f7-4332-9092-e7fef23819d2" containerName="revoke-keys" Nov 02 15:48:02.906339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.906284 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f977fedd-4641-483d-8c88-1fd88475c4b4" containerName="main" Nov 02 15:48:02.911070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.911052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:02.964703 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.964672 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s"] Nov 02 15:48:02.971997 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.971967 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:02.972159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.972007 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km26z\" (UniqueName: \"kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:02.972159 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:02.972098 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.007671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.007641 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:48:03.007862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.007846 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:48:03.072886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.072849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.072886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.072890 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km26z\" (UniqueName: \"kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.073098 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.072926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.073565 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.073544 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.075249 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.075225 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.131272 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.131238 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km26z\" (UniqueName: \"kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.224719 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.224639 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:03.577160 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.577131 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:48:03.579804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.579774 5091 generic.go:356] "Generic (PLEG): container finished" podID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerID="6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4" exitCode=0 Nov 02 15:48:03.579932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.579806 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" Nov 02 15:48:03.579932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.579807 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" event={"ID":"4746e31d-ebfd-4b9d-af84-9449d4690554","Type":"ContainerDied","Data":"6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4"} Nov 02 15:48:03.579932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.579859 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d" event={"ID":"4746e31d-ebfd-4b9d-af84-9449d4690554","Type":"ContainerDied","Data":"bee657a8f25331e2a050c07347b0024cf0829d7fee198c166799e961712e413b"} Nov 02 15:48:03.579932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.579884 5091 scope.go:122] "RemoveContainer" containerID="6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4" Nov 02 15:48:03.589069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.589050 5091 scope.go:122] "RemoveContainer" containerID="6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4" Nov 02 15:48:03.589362 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:03.589343 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4\": container with ID starting with 6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4 not found: ID does not exist" containerID="6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4" Nov 02 15:48:03.589428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.589377 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4"} err="failed to get container status \"6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4\": rpc error: code = NotFound desc = could not find container \"6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4\": container with ID starting with 6dd7c5508468e08bc03652bc57e988a0296a6b797fba51ec2288272d424fdcc4 not found: ID does not exist" Nov 02 15:48:03.598002 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.597976 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s"] Nov 02 15:48:03.602782 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:48:03.602754 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26a294f_aad3_4079_8563_735d82c607e7.slice/crio-e90f7b30fa6b26ddd072b92bed113bc007882d0995fa5149d3839f2ac7ff15fb WatchSource:0}: Error finding container e90f7b30fa6b26ddd072b92bed113bc007882d0995fa5149d3839f2ac7ff15fb: Status 404 returned error can't find the container with id e90f7b30fa6b26ddd072b92bed113bc007882d0995fa5149d3839f2ac7ff15fb Nov 02 15:48:03.678955 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.678923 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume\") pod \"4746e31d-ebfd-4b9d-af84-9449d4690554\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " Nov 02 15:48:03.679091 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.678963 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml\" (UniqueName: \"kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml\") pod \"4746e31d-ebfd-4b9d-af84-9449d4690554\" (UID: \"4746e31d-ebfd-4b9d-af84-9449d4690554\") " Nov 02 15:48:03.679362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.679340 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume" pod "4746e31d-ebfd-4b9d-af84-9449d4690554" (UID: "4746e31d-ebfd-4b9d-af84-9449d4690554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:03.679442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.679426 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4746e31d-ebfd-4b9d-af84-9449d4690554-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:03.681023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.681002 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml" pod "4746e31d-ebfd-4b9d-af84-9449d4690554" (UID: "4746e31d-ebfd-4b9d-af84-9449d4690554"). InnerVolumeSpecName "kube-api-access-c8sml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:03.780042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.779970 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8sml\" (UniqueName: \"kubernetes.io/projected/4746e31d-ebfd-4b9d-af84-9449d4690554-kube-api-access-c8sml\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:03.995062 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.995024 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d"] Nov 02 15:48:03.999964 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:03.999939 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-624908-6578ddc8c9-5vj6d"] Nov 02 15:48:04.588932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.588899 5091 generic.go:356] "Generic (PLEG): container finished" podID="b26a294f-aad3-4079-8563-735d82c607e7" containerID="47eaabfac1ed726e743becccd92e80fd7fc09829e5493c04eda3f5597285edd6" exitCode=0 Nov 02 15:48:04.589141 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.588986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" event={"ID":"b26a294f-aad3-4079-8563-735d82c607e7","Type":"ContainerDied","Data":"47eaabfac1ed726e743becccd92e80fd7fc09829e5493c04eda3f5597285edd6"} Nov 02 15:48:04.589141 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.589047 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" event={"ID":"b26a294f-aad3-4079-8563-735d82c607e7","Type":"ContainerStarted","Data":"e90f7b30fa6b26ddd072b92bed113bc007882d0995fa5149d3839f2ac7ff15fb"} Nov 02 15:48:04.617001 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.616966 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4"] Nov 02 15:48:04.617442 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.617429 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerName="payload-pre-processing" Nov 02 15:48:04.617497 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.617443 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerName="payload-pre-processing" Nov 02 15:48:04.617551 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.617530 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" containerName="payload-pre-processing" Nov 02 15:48:04.620661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.620642 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.678555 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.678496 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4"] Nov 02 15:48:04.678752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.678573 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq"] Nov 02 15:48:04.678752 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.678736 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" podUID="f65c0e09-68e3-4f91-847c-5175c4f18e78" containerName="maas-api" containerID="cri-o://93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f" gracePeriod=30 Nov 02 15:48:04.688857 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.688820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv2mn\" (UniqueName: \"kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.688993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.688860 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.689070 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.689053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.790295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.790192 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv2mn\" (UniqueName: \"kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.790295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.790246 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.790553 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.790395 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.791064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.791034 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.792960 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.792930 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.852076 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.851755 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv2mn\" (UniqueName: \"kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn\") pod \"maas-api-revoke-sub-eee8d285-k42x4\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.935000 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.934964 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:04.958996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:04.958966 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:48:05.094266 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.094237 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\") pod \"f65c0e09-68e3-4f91-847c-5175c4f18e78\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " Nov 02 15:48:05.094707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.094273 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt\" (UniqueName: \"kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt\") pod \"f65c0e09-68e3-4f91-847c-5175c4f18e78\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " Nov 02 15:48:05.094707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.094315 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs\") pod \"f65c0e09-68e3-4f91-847c-5175c4f18e78\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " Nov 02 15:48:05.094707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.094372 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca\") pod \"f65c0e09-68e3-4f91-847c-5175c4f18e78\" (UID: \"f65c0e09-68e3-4f91-847c-5175c4f18e78\") " Nov 02 15:48:05.094902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.094814 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca" pod "f65c0e09-68e3-4f91-847c-5175c4f18e78" (UID: "f65c0e09-68e3-4f91-847c-5175c4f18e78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:05.096506 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.096481 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs" pod "f65c0e09-68e3-4f91-847c-5175c4f18e78" (UID: "f65c0e09-68e3-4f91-847c-5175c4f18e78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:05.097033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.097013 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt" pod "f65c0e09-68e3-4f91-847c-5175c4f18e78" (UID: "f65c0e09-68e3-4f91-847c-5175c4f18e78"). InnerVolumeSpecName "kube-api-access-blrrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:05.097104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.097012 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls" pod "f65c0e09-68e3-4f91-847c-5175c4f18e78" (UID: "f65c0e09-68e3-4f91-847c-5175c4f18e78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:05.195605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.195566 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-maas-api-tls\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.195605 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.195611 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blrrt\" (UniqueName: \"kubernetes.io/projected/f65c0e09-68e3-4f91-847c-5175c4f18e78-kube-api-access-blrrt\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.195832 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.195627 5091 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f65c0e09-68e3-4f91-847c-5175c4f18e78-metrics-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.195832 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.195641 5091 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f65c0e09-68e3-4f91-847c-5175c4f18e78-openshift-service-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.316437 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.316401 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4"] Nov 02 15:48:05.321412 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:48:05.321386 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff65d5f_a424_4873_bd5e_849e883141de.slice/crio-89baf09fe357f962c3553ea209c862a376b3fd999c2097699c93f8baf1c1f27b WatchSource:0}: Error finding container 89baf09fe357f962c3553ea209c862a376b3fd999c2097699c93f8baf1c1f27b: Status 404 returned error can't find the container with id 89baf09fe357f962c3553ea209c862a376b3fd999c2097699c93f8baf1c1f27b Nov 02 15:48:05.597115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.597039 5091 generic.go:356] "Generic (PLEG): container finished" podID="f65c0e09-68e3-4f91-847c-5175c4f18e78" containerID="93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f" exitCode=0 Nov 02 15:48:05.597115 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.597105 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" Nov 02 15:48:05.597340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.597129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" event={"ID":"f65c0e09-68e3-4f91-847c-5175c4f18e78","Type":"ContainerDied","Data":"93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f"} Nov 02 15:48:05.597340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.597196 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq" event={"ID":"f65c0e09-68e3-4f91-847c-5175c4f18e78","Type":"ContainerDied","Data":"656fa444a33a86f5729309034de36e4900becd6c08360e1a4d0fc55bc918763d"} Nov 02 15:48:05.597340 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.597227 5091 scope.go:122] "RemoveContainer" containerID="93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f" Nov 02 15:48:05.598852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.598829 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" event={"ID":"7ff65d5f-a424-4873-bd5e-849e883141de","Type":"ContainerStarted","Data":"89baf09fe357f962c3553ea209c862a376b3fd999c2097699c93f8baf1c1f27b"} Nov 02 15:48:05.608344 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.608320 5091 scope.go:122] "RemoveContainer" containerID="93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f" Nov 02 15:48:05.608634 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:05.608614 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f\": container with ID starting with 93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f not found: ID does not exist" containerID="93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f" Nov 02 15:48:05.608710 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.608645 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f"} err="failed to get container status \"93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f\": rpc error: code = NotFound desc = could not find container \"93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f\": container with ID starting with 93f6f2ddfd82213102e4ba042969470ff8baecf19036aef6f0972908a0690e7f not found: ID does not exist" Nov 02 15:48:05.726281 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.726258 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:05.755044 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.755020 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq"] Nov 02 15:48:05.761371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.761348 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-624908-78c5986487-k9sfq"] Nov 02 15:48:05.802079 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.802057 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca\") pod \"b26a294f-aad3-4079-8563-735d82c607e7\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " Nov 02 15:48:05.802212 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.802193 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca\") pod \"b26a294f-aad3-4079-8563-735d82c607e7\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " Nov 02 15:48:05.802275 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.802257 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z\" (UniqueName: \"kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z\") pod \"b26a294f-aad3-4079-8563-735d82c607e7\" (UID: \"b26a294f-aad3-4079-8563-735d82c607e7\") " Nov 02 15:48:05.802462 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.802443 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca" pod "b26a294f-aad3-4079-8563-735d82c607e7" (UID: "b26a294f-aad3-4079-8563-735d82c607e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:05.802636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.802620 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.804171 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.804149 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca" pod "b26a294f-aad3-4079-8563-735d82c607e7" (UID: "b26a294f-aad3-4079-8563-735d82c607e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:05.804251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.804236 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z" pod "b26a294f-aad3-4079-8563-735d82c607e7" (UID: "b26a294f-aad3-4079-8563-735d82c607e7"). InnerVolumeSpecName "kube-api-access-km26z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:05.903216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.903183 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26a294f-aad3-4079-8563-735d82c607e7-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.903216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.903213 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km26z\" (UniqueName: \"kubernetes.io/projected/b26a294f-aad3-4079-8563-735d82c607e7-kube-api-access-km26z\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:05.959262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.959228 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4746e31d-ebfd-4b9d-af84-9449d4690554" path="/var/lib/kubelet/pods/4746e31d-ebfd-4b9d-af84-9449d4690554/volumes" Nov 02 15:48:05.959766 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:05.959745 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65c0e09-68e3-4f91-847c-5175c4f18e78" path="/var/lib/kubelet/pods/f65c0e09-68e3-4f91-847c-5175c4f18e78/volumes" Nov 02 15:48:06.606271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:06.606235 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" Nov 02 15:48:06.606271 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:06.606254 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s" event={"ID":"b26a294f-aad3-4079-8563-735d82c607e7","Type":"ContainerDied","Data":"e90f7b30fa6b26ddd072b92bed113bc007882d0995fa5149d3839f2ac7ff15fb"} Nov 02 15:48:06.606860 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:06.606309 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90f7b30fa6b26ddd072b92bed113bc007882d0995fa5149d3839f2ac7ff15fb" Nov 02 15:48:06.608023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:06.607996 5091 generic.go:356] "Generic (PLEG): container finished" podID="7ff65d5f-a424-4873-bd5e-849e883141de" containerID="f2e06c8e1235f7a852c8818d2797a57f62c4d55f13003fc04c02004ed800c67b" exitCode=0 Nov 02 15:48:06.608243 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:06.608093 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" event={"ID":"7ff65d5f-a424-4873-bd5e-849e883141de","Type":"ContainerDied","Data":"f2e06c8e1235f7a852c8818d2797a57f62c4d55f13003fc04c02004ed800c67b"} Nov 02 15:48:07.143583 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.143548 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms"] Nov 02 15:48:07.146654 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.146629 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-624908-796ms"] Nov 02 15:48:07.780988 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.780963 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:07.923105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.923026 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca\") pod \"7ff65d5f-a424-4873-bd5e-849e883141de\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " Nov 02 15:48:07.923105 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.923086 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn\" (UniqueName: \"kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn\") pod \"7ff65d5f-a424-4873-bd5e-849e883141de\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " Nov 02 15:48:07.923352 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.923137 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca\") pod \"7ff65d5f-a424-4873-bd5e-849e883141de\" (UID: \"7ff65d5f-a424-4873-bd5e-849e883141de\") " Nov 02 15:48:07.923608 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.923580 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca" pod "7ff65d5f-a424-4873-bd5e-849e883141de" (UID: "7ff65d5f-a424-4873-bd5e-849e883141de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:07.925270 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.925237 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca" pod "7ff65d5f-a424-4873-bd5e-849e883141de" (UID: "7ff65d5f-a424-4873-bd5e-849e883141de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:07.925371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.925277 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn" pod "7ff65d5f-a424-4873-bd5e-849e883141de" (UID: "7ff65d5f-a424-4873-bd5e-849e883141de"). InnerVolumeSpecName "kube-api-access-cv2mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:07.958983 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:07.958951 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac8fc47-98b1-4a70-a545-77a6ffb35ae7" path="/var/lib/kubelet/pods/4ac8fc47-98b1-4a70-a545-77a6ffb35ae7/volumes" Nov 02 15:48:08.025069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:08.025027 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv2mn\" (UniqueName: \"kubernetes.io/projected/7ff65d5f-a424-4873-bd5e-849e883141de-kube-api-access-cv2mn\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:08.025069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:08.025061 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:08.025069 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:08.025072 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff65d5f-a424-4873-bd5e-849e883141de-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:08.622912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:08.622879 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" Nov 02 15:48:08.622912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:08.622910 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4" event={"ID":"7ff65d5f-a424-4873-bd5e-849e883141de","Type":"ContainerDied","Data":"89baf09fe357f962c3553ea209c862a376b3fd999c2097699c93f8baf1c1f27b"} Nov 02 15:48:08.623153 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:08.622962 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89baf09fe357f962c3553ea209c862a376b3fd999c2097699c93f8baf1c1f27b" Nov 02 15:48:13.841844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:13.841804 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh"] Nov 02 15:48:13.889350 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:13.889322 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-e98976-jc6sh"] Nov 02 15:48:13.958899 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:13.958862 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87793f1e-4daf-45b3-beee-969b101950c3" path="/var/lib/kubelet/pods/87793f1e-4daf-45b3-beee-969b101950c3/volumes" Nov 02 15:48:15.060297 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:15.060263 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58"] Nov 02 15:48:15.149452 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:15.149415 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf"] Nov 02 15:48:20.060772 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.060701 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" podUID="aa336170-94a8-42dd-be65-8b294552805f" containerName="payload-processing" containerID="cri-o://4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d" gracePeriod=25 Nov 02 15:48:20.149773 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.149734 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" podUID="62fb6960-de96-4379-adb3-0a014817beaa" containerName="payload-pre-processing" containerID="cri-o://b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792" gracePeriod=25 Nov 02 15:48:20.413218 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.413191 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:48:20.447770 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.447745 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume\") pod \"aa336170-94a8-42dd-be65-8b294552805f\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " Nov 02 15:48:20.447907 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.447836 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s\" (UniqueName: \"kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s\") pod \"aa336170-94a8-42dd-be65-8b294552805f\" (UID: \"aa336170-94a8-42dd-be65-8b294552805f\") " Nov 02 15:48:20.448164 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.448140 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume" pod "aa336170-94a8-42dd-be65-8b294552805f" (UID: "aa336170-94a8-42dd-be65-8b294552805f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:20.450037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.450016 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s" pod "aa336170-94a8-42dd-be65-8b294552805f" (UID: "aa336170-94a8-42dd-be65-8b294552805f"). InnerVolumeSpecName "kube-api-access-jnd2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:20.452351 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.452328 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:48:20.549080 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.549047 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume\") pod \"62fb6960-de96-4379-adb3-0a014817beaa\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " Nov 02 15:48:20.549247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.549094 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2\" (UniqueName: \"kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2\") pod \"62fb6960-de96-4379-adb3-0a014817beaa\" (UID: \"62fb6960-de96-4379-adb3-0a014817beaa\") " Nov 02 15:48:20.549319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.549307 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa336170-94a8-42dd-be65-8b294552805f-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:20.549354 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.549321 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnd2s\" (UniqueName: \"kubernetes.io/projected/aa336170-94a8-42dd-be65-8b294552805f-kube-api-access-jnd2s\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:20.549568 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.549506 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume" pod "62fb6960-de96-4379-adb3-0a014817beaa" (UID: "62fb6960-de96-4379-adb3-0a014817beaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:20.551255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.551212 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2" pod "62fb6960-de96-4379-adb3-0a014817beaa" (UID: "62fb6960-de96-4379-adb3-0a014817beaa"). InnerVolumeSpecName "kube-api-access-jxcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:20.577255 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.577192 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s"] Nov 02 15:48:20.580056 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.580034 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7ad552-msf9s"] Nov 02 15:48:20.650431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.650399 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62fb6960-de96-4379-adb3-0a014817beaa-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:20.650431 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.650427 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxcs2\" (UniqueName: \"kubernetes.io/projected/62fb6960-de96-4379-adb3-0a014817beaa-kube-api-access-jxcs2\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:20.700161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.699798 5091 generic.go:356] "Generic (PLEG): container finished" podID="aa336170-94a8-42dd-be65-8b294552805f" containerID="4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d" exitCode=0 Nov 02 15:48:20.700161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.699856 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" Nov 02 15:48:20.700161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.699951 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" event={"ID":"aa336170-94a8-42dd-be65-8b294552805f","Type":"ContainerDied","Data":"4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d"} Nov 02 15:48:20.700161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.700000 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58" event={"ID":"aa336170-94a8-42dd-be65-8b294552805f","Type":"ContainerDied","Data":"a358cb7645789ccf2acf791128c81253d136ba88f16c87331d18558ce7a4c136"} Nov 02 15:48:20.700161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.700029 5091 scope.go:122] "RemoveContainer" containerID="4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d" Nov 02 15:48:20.704170 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.704144 5091 generic.go:356] "Generic (PLEG): container finished" podID="62fb6960-de96-4379-adb3-0a014817beaa" containerID="b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792" exitCode=0 Nov 02 15:48:20.704474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.704192 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" Nov 02 15:48:20.704961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.704211 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" event={"ID":"62fb6960-de96-4379-adb3-0a014817beaa","Type":"ContainerDied","Data":"b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792"} Nov 02 15:48:20.705057 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.704996 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf" event={"ID":"62fb6960-de96-4379-adb3-0a014817beaa","Type":"ContainerDied","Data":"0cb4ec3f6e5b73048da25200306e89652a10ebbcdc070d7f345a89d9e8a8dc5d"} Nov 02 15:48:20.715147 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.715118 5091 scope.go:122] "RemoveContainer" containerID="4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d" Nov 02 15:48:20.715985 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:20.715955 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d\": container with ID starting with 4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d not found: ID does not exist" containerID="4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d" Nov 02 15:48:20.716116 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.716000 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d"} err="failed to get container status \"4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d\": rpc error: code = NotFound desc = could not find container \"4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d\": container with ID starting with 4c6659500455499c77c021c805ea88974eee96af9e0cf2cf579128cf7235479d not found: ID does not exist" Nov 02 15:48:20.716116 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.716025 5091 scope.go:122] "RemoveContainer" containerID="b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792" Nov 02 15:48:20.735117 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.734811 5091 scope.go:122] "RemoveContainer" containerID="b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792" Nov 02 15:48:20.735258 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:20.735184 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792\": container with ID starting with b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792 not found: ID does not exist" containerID="b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792" Nov 02 15:48:20.735258 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.735213 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792"} err="failed to get container status \"b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792\": rpc error: code = NotFound desc = could not find container \"b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792\": container with ID starting with b372bd343ec929aba0afecfb620afdd2fbaaa5729e76c3d3bd8a24f06151d792 not found: ID does not exist" Nov 02 15:48:20.838470 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.838379 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t"] Nov 02 15:48:20.838966 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.838928 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" containerID="cri-o://5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73" gracePeriod=30 Nov 02 15:48:20.861554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.861490 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf"] Nov 02 15:48:20.865256 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.865236 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7ad552-795cf759f5-wxtgf"] Nov 02 15:48:20.965586 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.965547 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58"] Nov 02 15:48:20.970763 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:20.970726 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7ad552-849c768b6f-t7r58"] Nov 02 15:48:21.726222 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:21.726187 5091 patch_prober.go:29] interesting pod/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 15:48:21.726639 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:21.726242 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 15:48:21.959148 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:21.959109 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62fb6960-de96-4379-adb3-0a014817beaa" path="/var/lib/kubelet/pods/62fb6960-de96-4379-adb3-0a014817beaa/volumes" Nov 02 15:48:21.959557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:21.959542 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa336170-94a8-42dd-be65-8b294552805f" path="/var/lib/kubelet/pods/aa336170-94a8-42dd-be65-8b294552805f/volumes" Nov 02 15:48:21.959905 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:21.959894 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26a294f-aad3-4079-8563-735d82c607e7" path="/var/lib/kubelet/pods/b26a294f-aad3-4079-8563-735d82c607e7/volumes" Nov 02 15:48:26.097152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.097129 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:48:26.205478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205445 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.205671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205485 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.205671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205553 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.205671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205578 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.205671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205596 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.205903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205698 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.205903 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205738 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.206089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206071 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.206187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.205792 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:26.206187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:26.206187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206033 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:26.206187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206115 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket\") pod \"e1391062-b3d8-4c72-882b-c1d4b765275d\" (UID: \"e1391062-b3d8-4c72-882b-c1d4b765275d\") " Nov 02 15:48:26.206401 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206305 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:26.206401 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206317 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:26.206554 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206538 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.206619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206559 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.206619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206574 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.206619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206588 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.206619 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.206604 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1391062-b3d8-4c72-882b-c1d4b765275d-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.207992 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.207965 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:26.208099 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.208001 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "kube-api-access-9hvt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:26.208099 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.208013 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:48:26.208099 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.208046 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy" pod "e1391062-b3d8-4c72-882b-c1d4b765275d" (UID: "e1391062-b3d8-4c72-882b-c1d4b765275d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:48:26.307967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.307935 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.307967 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.307964 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.308178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.307981 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hvt8\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-kube-api-access-9hvt8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.308178 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.307995 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1391062-b3d8-4c72-882b-c1d4b765275d-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:26.746977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.746936 5091 generic.go:356] "Generic (PLEG): container finished" podID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerID="5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73" exitCode=0 Nov 02 15:48:26.747161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.746990 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" Nov 02 15:48:26.747161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.747025 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" event={"ID":"e1391062-b3d8-4c72-882b-c1d4b765275d","Type":"ContainerDied","Data":"5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73"} Nov 02 15:48:26.747161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.747083 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t" event={"ID":"e1391062-b3d8-4c72-882b-c1d4b765275d","Type":"ContainerDied","Data":"5ba6c9c3e7715b786d25a1c8ca38a1d2890cbed795aa45fb51e4bdabda8b51a0"} Nov 02 15:48:26.747161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.747106 5091 scope.go:122] "RemoveContainer" containerID="5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73" Nov 02 15:48:26.759369 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.759353 5091 scope.go:122] "RemoveContainer" containerID="5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73" Nov 02 15:48:26.759630 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:26.759607 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73\": container with ID starting with 5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73 not found: ID does not exist" containerID="5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73" Nov 02 15:48:26.759683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.759644 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73"} err="failed to get container status \"5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73\": rpc error: code = NotFound desc = could not find container \"5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73\": container with ID starting with 5a617ebc27cb3a1b366208b97561d15eddea49c26e8ea9b8d062520b0ad8ac73 not found: ID does not exist" Nov 02 15:48:26.903757 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.903723 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t"] Nov 02 15:48:26.950602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.950568 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-e98976-openshift-default-859bd7b549-bwx7t"] Nov 02 15:48:26.994422 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.994391 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq"] Nov 02 15:48:26.998319 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:26.998261 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8a2e6f-cr2rq"] Nov 02 15:48:27.936765 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:27.936719 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:48:27.960058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:27.960017 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4d9739-e9a2-43c2-8731-42e9e7aff7a1" path="/var/lib/kubelet/pods/9e4d9739-e9a2-43c2-8731-42e9e7aff7a1/volumes" Nov 02 15:48:27.960538 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:27.960503 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" path="/var/lib/kubelet/pods/e1391062-b3d8-4c72-882b-c1d4b765275d/volumes" Nov 02 15:48:28.743448 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743409 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv"] Nov 02 15:48:28.743854 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743838 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743855 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743873 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7ff65d5f-a424-4873-bd5e-849e883141de" containerName="revoke-keys" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743878 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff65d5f-a424-4873-bd5e-849e883141de" containerName="revoke-keys" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743887 5091 cpu_manager.go:414] "containerMap: removing container" podUID="aa336170-94a8-42dd-be65-8b294552805f" containerName="payload-processing" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743892 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa336170-94a8-42dd-be65-8b294552805f" containerName="payload-processing" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743905 5091 cpu_manager.go:414] "containerMap: removing container" podUID="62fb6960-de96-4379-adb3-0a014817beaa" containerName="payload-pre-processing" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743910 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fb6960-de96-4379-adb3-0a014817beaa" containerName="payload-pre-processing" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743925 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b26a294f-aad3-4079-8563-735d82c607e7" containerName="revoke-keys" Nov 02 15:48:28.743928 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743930 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26a294f-aad3-4079-8563-735d82c607e7" containerName="revoke-keys" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743946 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f65c0e09-68e3-4f91-847c-5175c4f18e78" containerName="maas-api" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.743951 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65c0e09-68e3-4f91-847c-5175c4f18e78" containerName="maas-api" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.744022 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fb6960-de96-4379-adb3-0a014817beaa" containerName="payload-pre-processing" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.744032 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26a294f-aad3-4079-8563-735d82c607e7" containerName="revoke-keys" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.744041 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65c0e09-68e3-4f91-847c-5175c4f18e78" containerName="maas-api" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.744047 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1391062-b3d8-4c72-882b-c1d4b765275d" containerName="istio-proxy" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.744056 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff65d5f-a424-4873-bd5e-849e883141de" containerName="revoke-keys" Nov 02 15:48:28.744214 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.744062 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa336170-94a8-42dd-be65-8b294552805f" containerName="payload-processing" Nov 02 15:48:28.748697 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.748675 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.799876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.799832 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:48:28.833014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.832980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.833124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.833103 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxnz\" (UniqueName: \"kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.833176 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.833151 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.846375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.846357 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:48:28.848659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.848641 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv"] Nov 02 15:48:28.893708 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.893688 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:48:28.893820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.893741 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:48:28.934139 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.934114 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.934251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.934162 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxnz\" (UniqueName: \"kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.934296 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.934275 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.934806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.934789 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.936676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.936656 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:28.994120 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:28.994047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxnz\" (UniqueName: \"kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:29.062379 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:29.062357 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:29.435009 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:29.434971 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv"] Nov 02 15:48:29.441487 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:48:29.441458 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e0d32d_f0e8_4a33_9b21_0c3d442512d8.slice/crio-bd08fe5080d3b345ace88f004219ce6448af42274591e1da31f49b39074243be WatchSource:0}: Error finding container bd08fe5080d3b345ace88f004219ce6448af42274591e1da31f49b39074243be: Status 404 returned error can't find the container with id bd08fe5080d3b345ace88f004219ce6448af42274591e1da31f49b39074243be Nov 02 15:48:29.770530 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:29.770494 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" event={"ID":"23e0d32d-f0e8-4a33-9b21-0c3d442512d8","Type":"ContainerStarted","Data":"e21509aa6216c5990d1d16f18d972b4d912becfe4401575ae882b3be5fde9aed"} Nov 02 15:48:29.770636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:29.770556 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" event={"ID":"23e0d32d-f0e8-4a33-9b21-0c3d442512d8","Type":"ContainerStarted","Data":"bd08fe5080d3b345ace88f004219ce6448af42274591e1da31f49b39074243be"} Nov 02 15:48:29.883193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:29.883141 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" podStartSLOduration=1.883121692 podStartE2EDuration="1.883121692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:48:29.88085459 +0000 UTC m=+3064.045407131" watchObservedRunningTime="2026-11-02 15:48:29.883121692 +0000 UTC m=+3064.047674231" Nov 02 15:48:30.254678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:30.254645 5091 scope.go:122] "RemoveContainer" containerID="504dcefd932d6db7fdaf72553ad0a38b4fc78d143ccb085f99b628bdacf47b0c" Nov 02 15:48:30.264126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:30.264104 5091 scope.go:122] "RemoveContainer" containerID="595b6a5f070f844c78442bd9e5b4f29d872d0b5b2812de120724eae81ca11e6a" Nov 02 15:48:30.778908 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:30.778871 5091 generic.go:356] "Generic (PLEG): container finished" podID="23e0d32d-f0e8-4a33-9b21-0c3d442512d8" containerID="e21509aa6216c5990d1d16f18d972b4d912becfe4401575ae882b3be5fde9aed" exitCode=0 Nov 02 15:48:30.779087 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:30.778958 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" event={"ID":"23e0d32d-f0e8-4a33-9b21-0c3d442512d8","Type":"ContainerDied","Data":"e21509aa6216c5990d1d16f18d972b4d912becfe4401575ae882b3be5fde9aed"} Nov 02 15:48:31.931213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.931187 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:31.966014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.965988 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca\") pod \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " Nov 02 15:48:31.966135 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.966078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca\") pod \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " Nov 02 15:48:31.966187 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.966176 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz\" (UniqueName: \"kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz\") pod \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\" (UID: \"23e0d32d-f0e8-4a33-9b21-0c3d442512d8\") " Nov 02 15:48:31.966491 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.966460 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca" pod "23e0d32d-f0e8-4a33-9b21-0c3d442512d8" (UID: "23e0d32d-f0e8-4a33-9b21-0c3d442512d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:48:31.968077 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.968058 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca" pod "23e0d32d-f0e8-4a33-9b21-0c3d442512d8" (UID: "23e0d32d-f0e8-4a33-9b21-0c3d442512d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:48:31.968232 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:31.968212 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz" pod "23e0d32d-f0e8-4a33-9b21-0c3d442512d8" (UID: "23e0d32d-f0e8-4a33-9b21-0c3d442512d8"). InnerVolumeSpecName "kube-api-access-2wxnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:48:32.067912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:32.067828 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:32.067912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:32.067860 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wxnz\" (UniqueName: \"kubernetes.io/projected/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-kube-api-access-2wxnz\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:32.067912 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:32.067873 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e0d32d-f0e8-4a33-9b21-0c3d442512d8-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:48:32.795500 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:32.795453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" event={"ID":"23e0d32d-f0e8-4a33-9b21-0c3d442512d8","Type":"ContainerDied","Data":"bd08fe5080d3b345ace88f004219ce6448af42274591e1da31f49b39074243be"} Nov 02 15:48:32.795847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:32.795558 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv" Nov 02 15:48:32.795847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:32.795565 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd08fe5080d3b345ace88f004219ce6448af42274591e1da31f49b39074243be" Nov 02 15:48:54.067211 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:54.067174 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv"] Nov 02 15:48:54.113798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:54.113765 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ff6e1f-g7jjv"] Nov 02 15:48:55.958578 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:48:55.958507 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e0d32d-f0e8-4a33-9b21-0c3d442512d8" path="/var/lib/kubelet/pods/23e0d32d-f0e8-4a33-9b21-0c3d442512d8/volumes" Nov 02 15:48:57.937525 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:48:57.937469 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:49:06.673096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:06.673018 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5"] Nov 02 15:49:06.720175 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:06.720144 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc"] Nov 02 15:49:06.722126 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:06.722097 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27"] Nov 02 15:49:14.151740 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.151703 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x"] Nov 02 15:49:14.152377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.152343 5091 cpu_manager.go:414] "containerMap: removing container" podUID="23e0d32d-f0e8-4a33-9b21-0c3d442512d8" containerName="revoke-keys" Nov 02 15:49:14.152377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.152361 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e0d32d-f0e8-4a33-9b21-0c3d442512d8" containerName="revoke-keys" Nov 02 15:49:14.152536 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.152468 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e0d32d-f0e8-4a33-9b21-0c3d442512d8" containerName="revoke-keys" Nov 02 15:49:14.156294 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.156273 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.240643 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.240615 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:49:14.253574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.253546 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:49:14.253574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.253561 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:49:14.253574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.253553 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:49:14.268291 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.268260 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglqn\" (UniqueName: \"kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.268428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.268318 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.268428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.268401 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.369707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.369673 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.369707 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.369711 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.369941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.369772 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglqn\" (UniqueName: \"kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.370585 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.370556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.372368 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.372344 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x"] Nov 02 15:49:14.372659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.372640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.428327 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.428266 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglqn\" (UniqueName: \"kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn\") pod \"maas-api-revoke-sub-01b9ad66-q948x\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.469659 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.469634 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:14.839235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:14.839163 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x"] Nov 02 15:49:14.844576 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:49:14.844543 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd161fc_0957_41d4_8acf_0d0be7275149.slice/crio-73e62aa5e803c95ca7250fa0ba885a1d6040ead37d6b9f763aaa59e02c94b133 WatchSource:0}: Error finding container 73e62aa5e803c95ca7250fa0ba885a1d6040ead37d6b9f763aaa59e02c94b133: Status 404 returned error can't find the container with id 73e62aa5e803c95ca7250fa0ba885a1d6040ead37d6b9f763aaa59e02c94b133 Nov 02 15:49:15.043358 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:15.043324 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" event={"ID":"1fd161fc-0957-41d4-8acf-0d0be7275149","Type":"ContainerStarted","Data":"73e62aa5e803c95ca7250fa0ba885a1d6040ead37d6b9f763aaa59e02c94b133"} Nov 02 15:49:16.050545 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:16.050493 5091 generic.go:356] "Generic (PLEG): container finished" podID="1fd161fc-0957-41d4-8acf-0d0be7275149" containerID="15884abd508eb02138746203d34858d1342a660441192f46748004c2f1bfb73b" exitCode=0 Nov 02 15:49:16.050957 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:16.050573 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" event={"ID":"1fd161fc-0957-41d4-8acf-0d0be7275149","Type":"ContainerDied","Data":"15884abd508eb02138746203d34858d1342a660441192f46748004c2f1bfb73b"} Nov 02 15:49:17.200842 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.200811 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:17.296733 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.296700 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca\") pod \"1fd161fc-0957-41d4-8acf-0d0be7275149\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " Nov 02 15:49:17.296917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.296767 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn\" (UniqueName: \"kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn\") pod \"1fd161fc-0957-41d4-8acf-0d0be7275149\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " Nov 02 15:49:17.296917 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.296838 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca\") pod \"1fd161fc-0957-41d4-8acf-0d0be7275149\" (UID: \"1fd161fc-0957-41d4-8acf-0d0be7275149\") " Nov 02 15:49:17.297196 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.297172 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca" pod "1fd161fc-0957-41d4-8acf-0d0be7275149" (UID: "1fd161fc-0957-41d4-8acf-0d0be7275149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:17.298782 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.298760 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca" pod "1fd161fc-0957-41d4-8acf-0d0be7275149" (UID: "1fd161fc-0957-41d4-8acf-0d0be7275149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:17.298871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.298820 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn" pod "1fd161fc-0957-41d4-8acf-0d0be7275149" (UID: "1fd161fc-0957-41d4-8acf-0d0be7275149"). InnerVolumeSpecName "kube-api-access-sglqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:17.397781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.397751 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:17.397781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.397778 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sglqn\" (UniqueName: \"kubernetes.io/projected/1fd161fc-0957-41d4-8acf-0d0be7275149-kube-api-access-sglqn\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:17.397993 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:17.397788 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd161fc-0957-41d4-8acf-0d0be7275149-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:18.065361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:18.065333 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" Nov 02 15:49:18.065361 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:18.065347 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x" event={"ID":"1fd161fc-0957-41d4-8acf-0d0be7275149","Type":"ContainerDied","Data":"73e62aa5e803c95ca7250fa0ba885a1d6040ead37d6b9f763aaa59e02c94b133"} Nov 02 15:49:18.065610 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:18.065389 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e62aa5e803c95ca7250fa0ba885a1d6040ead37d6b9f763aaa59e02c94b133" Nov 02 15:49:21.693719 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:21.693672 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" containerID="cri-o://006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419" gracePeriod=45 Nov 02 15:49:21.740066 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:21.740004 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" containerID="cri-o://87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f" gracePeriod=45 Nov 02 15:49:21.740812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:21.740761 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" containerID="cri-o://e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722" gracePeriod=45 Nov 02 15:49:21.790199 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:21.790165 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": dial tcp 10.132.0.62:8000: connect: connection refused" Nov 02 15:49:22.032453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.032419 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:49:22.054047 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.054019 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:49:22.073899 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.073870 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:49:22.089262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.089236 5091 generic.go:356] "Generic (PLEG): container finished" podID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerID="87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f" exitCode=0 Nov 02 15:49:22.089411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.089307 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" event={"ID":"a8bf04d7-e230-42d5-ad2a-d087d2611533","Type":"ContainerDied","Data":"87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f"} Nov 02 15:49:22.089411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.089332 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" event={"ID":"a8bf04d7-e230-42d5-ad2a-d087d2611533","Type":"ContainerDied","Data":"9555170abcf0584f78f33a54d728902279f02b3e3773e28f0a51029f901dec36"} Nov 02 15:49:22.089411 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.089349 5091 scope.go:122] "RemoveContainer" containerID="87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f" Nov 02 15:49:22.089618 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.089447 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc" Nov 02 15:49:22.091121 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.091098 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerID="006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419" exitCode=0 Nov 02 15:49:22.091261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.091150 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" Nov 02 15:49:22.091339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.091278 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" event={"ID":"fc688f54-c7e1-4244-9a54-c47dc066520a","Type":"ContainerDied","Data":"006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419"} Nov 02 15:49:22.091339 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.091307 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5" event={"ID":"fc688f54-c7e1-4244-9a54-c47dc066520a","Type":"ContainerDied","Data":"e30641e19139463c291f7441f947c31b385126a47a85a87b195d7bc590ad9f7a"} Nov 02 15:49:22.092852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.092829 5091 generic.go:356] "Generic (PLEG): container finished" podID="6b9b569c-2c19-48e7-bb05-109719649670" containerID="e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722" exitCode=0 Nov 02 15:49:22.092979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.092864 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" Nov 02 15:49:22.092979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.092879 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" event={"ID":"6b9b569c-2c19-48e7-bb05-109719649670","Type":"ContainerDied","Data":"e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722"} Nov 02 15:49:22.092979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.092906 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27" event={"ID":"6b9b569c-2c19-48e7-bb05-109719649670","Type":"ContainerDied","Data":"faf4995c3bd85cedcf717a522b8f38ed68c09e7e91a25a1ca16ba135ad0c07f1"} Nov 02 15:49:22.110495 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.110469 5091 scope.go:122] "RemoveContainer" containerID="87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f" Nov 02 15:49:22.110819 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:49:22.110792 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f\": container with ID starting with 87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f not found: ID does not exist" containerID="87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f" Nov 02 15:49:22.110884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.110833 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f"} err="failed to get container status \"87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f\": rpc error: code = NotFound desc = could not find container \"87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f\": container with ID starting with 87ec02a9262a62235df80e054fcfedd7e7f92eb62d2d6199a50a0202dad8a74f not found: ID does not exist" Nov 02 15:49:22.110884 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.110860 5091 scope.go:122] "RemoveContainer" containerID="006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419" Nov 02 15:49:22.121661 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.121630 5091 scope.go:122] "RemoveContainer" containerID="006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419" Nov 02 15:49:22.121927 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:49:22.121907 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419\": container with ID starting with 006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419 not found: ID does not exist" containerID="006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419" Nov 02 15:49:22.121999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.121939 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419"} err="failed to get container status \"006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419\": rpc error: code = NotFound desc = could not find container \"006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419\": container with ID starting with 006757b84637556c0436e22180284b63bb498fe0180d99219776747a5e6f8419 not found: ID does not exist" Nov 02 15:49:22.121999 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.121965 5091 scope.go:122] "RemoveContainer" containerID="e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722" Nov 02 15:49:22.132377 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.132353 5091 scope.go:122] "RemoveContainer" containerID="e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722" Nov 02 15:49:22.132727 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:49:22.132704 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722\": container with ID starting with e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722 not found: ID does not exist" containerID="e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722" Nov 02 15:49:22.132812 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.132740 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722"} err="failed to get container status \"e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722\": rpc error: code = NotFound desc = could not find container \"e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722\": container with ID starting with e5b0df05bd532360369d336cc27552c8a9cd3b43f85b4e7dd9fc93beb8f02722 not found: ID does not exist" Nov 02 15:49:22.143995 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.143975 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home\") pod \"6b9b569c-2c19-48e7-bb05-109719649670\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " Nov 02 15:49:22.144096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144009 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm\") pod \"fc688f54-c7e1-4244-9a54-c47dc066520a\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " Nov 02 15:49:22.144096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144033 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs\") pod \"fc688f54-c7e1-4244-9a54-c47dc066520a\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " Nov 02 15:49:22.144096 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs\") pod \"6b9b569c-2c19-48e7-bb05-109719649670\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " Nov 02 15:49:22.144244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144094 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache\") pod \"6b9b569c-2c19-48e7-bb05-109719649670\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " Nov 02 15:49:22.144244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144135 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4\" (UniqueName: \"kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4\") pod \"fc688f54-c7e1-4244-9a54-c47dc066520a\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " Nov 02 15:49:22.144244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144170 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp\" (UniqueName: \"kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp\") pod \"a8bf04d7-e230-42d5-ad2a-d087d2611533\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " Nov 02 15:49:22.144244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144201 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir\") pod \"a8bf04d7-e230-42d5-ad2a-d087d2611533\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " Nov 02 15:49:22.144466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144286 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc\" (UniqueName: \"kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc\") pod \"6b9b569c-2c19-48e7-bb05-109719649670\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " Nov 02 15:49:22.144466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144316 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home\") pod \"a8bf04d7-e230-42d5-ad2a-d087d2611533\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " Nov 02 15:49:22.144466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144349 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home" pod "6b9b569c-2c19-48e7-bb05-109719649670" (UID: "6b9b569c-2c19-48e7-bb05-109719649670"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.144466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144379 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir\") pod \"fc688f54-c7e1-4244-9a54-c47dc066520a\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " Nov 02 15:49:22.144466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir\") pod \"6b9b569c-2c19-48e7-bb05-109719649670\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " Nov 02 15:49:22.144466 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144460 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm\") pod \"a8bf04d7-e230-42d5-ad2a-d087d2611533\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " Nov 02 15:49:22.145067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144614 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home\") pod \"fc688f54-c7e1-4244-9a54-c47dc066520a\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " Nov 02 15:49:22.145067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144642 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache\") pod \"fc688f54-c7e1-4244-9a54-c47dc066520a\" (UID: \"fc688f54-c7e1-4244-9a54-c47dc066520a\") " Nov 02 15:49:22.145067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144692 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs\") pod \"a8bf04d7-e230-42d5-ad2a-d087d2611533\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " Nov 02 15:49:22.145067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144732 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm\") pod \"6b9b569c-2c19-48e7-bb05-109719649670\" (UID: \"6b9b569c-2c19-48e7-bb05-109719649670\") " Nov 02 15:49:22.145067 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.144784 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache\") pod \"a8bf04d7-e230-42d5-ad2a-d087d2611533\" (UID: \"a8bf04d7-e230-42d5-ad2a-d087d2611533\") " Nov 02 15:49:22.145480 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.145410 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.145924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.145899 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache" pod "a8bf04d7-e230-42d5-ad2a-d087d2611533" (UID: "a8bf04d7-e230-42d5-ad2a-d087d2611533"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.146180 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.146158 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home" pod "a8bf04d7-e230-42d5-ad2a-d087d2611533" (UID: "a8bf04d7-e230-42d5-ad2a-d087d2611533"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.146410 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.146391 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir" pod "fc688f54-c7e1-4244-9a54-c47dc066520a" (UID: "fc688f54-c7e1-4244-9a54-c47dc066520a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.146683 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.146645 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir" pod "6b9b569c-2c19-48e7-bb05-109719649670" (UID: "6b9b569c-2c19-48e7-bb05-109719649670"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.148064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.147830 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache" pod "6b9b569c-2c19-48e7-bb05-109719649670" (UID: "6b9b569c-2c19-48e7-bb05-109719649670"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.148064 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.147887 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir" pod "a8bf04d7-e230-42d5-ad2a-d087d2611533" (UID: "a8bf04d7-e230-42d5-ad2a-d087d2611533"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.148443 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.148268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache" pod "fc688f54-c7e1-4244-9a54-c47dc066520a" (UID: "fc688f54-c7e1-4244-9a54-c47dc066520a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.148808 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.148594 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home" pod "fc688f54-c7e1-4244-9a54-c47dc066520a" (UID: "fc688f54-c7e1-4244-9a54-c47dc066520a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.151293 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.151270 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm" pod "fc688f54-c7e1-4244-9a54-c47dc066520a" (UID: "fc688f54-c7e1-4244-9a54-c47dc066520a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.153061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.152959 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc" pod "6b9b569c-2c19-48e7-bb05-109719649670" (UID: "6b9b569c-2c19-48e7-bb05-109719649670"). InnerVolumeSpecName "kube-api-access-dvdgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:22.153596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.153454 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4" pod "fc688f54-c7e1-4244-9a54-c47dc066520a" (UID: "fc688f54-c7e1-4244-9a54-c47dc066520a"). InnerVolumeSpecName "kube-api-access-k4sj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:22.153596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.153541 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs" pod "6b9b569c-2c19-48e7-bb05-109719649670" (UID: "6b9b569c-2c19-48e7-bb05-109719649670"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:22.153596 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.153569 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs" pod "a8bf04d7-e230-42d5-ad2a-d087d2611533" (UID: "a8bf04d7-e230-42d5-ad2a-d087d2611533"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:22.153781 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.153687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs" pod "fc688f54-c7e1-4244-9a54-c47dc066520a" (UID: "fc688f54-c7e1-4244-9a54-c47dc066520a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:49:22.154396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.154371 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp" pod "a8bf04d7-e230-42d5-ad2a-d087d2611533" (UID: "a8bf04d7-e230-42d5-ad2a-d087d2611533"). InnerVolumeSpecName "kube-api-access-ntwtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:22.154806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.154784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm" pod "6b9b569c-2c19-48e7-bb05-109719649670" (UID: "6b9b569c-2c19-48e7-bb05-109719649670"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.155161 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.155122 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm" pod "a8bf04d7-e230-42d5-ad2a-d087d2611533" (UID: "a8bf04d7-e230-42d5-ad2a-d087d2611533"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246566 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdgc\" (UniqueName: \"kubernetes.io/projected/6b9b569c-2c19-48e7-bb05-109719649670-kube-api-access-dvdgc\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246597 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246607 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246615 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246623 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246632 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246640 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.246645 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246649 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bf04d7-e230-42d5-ad2a-d087d2611533-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246659 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246671 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246679 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc688f54-c7e1-4244-9a54-c47dc066520a-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246688 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc688f54-c7e1-4244-9a54-c47dc066520a-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246695 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b9b569c-2c19-48e7-bb05-109719649670-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246704 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b9b569c-2c19-48e7-bb05-109719649670-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246712 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4sj4\" (UniqueName: \"kubernetes.io/projected/fc688f54-c7e1-4244-9a54-c47dc066520a-kube-api-access-k4sj4\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246721 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntwtp\" (UniqueName: \"kubernetes.io/projected/a8bf04d7-e230-42d5-ad2a-d087d2611533-kube-api-access-ntwtp\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.247026 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.246730 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bf04d7-e230-42d5-ad2a-d087d2611533-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:22.531841 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.531760 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc"] Nov 02 15:49:22.535681 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.535655 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/premium-w2-fce776-kserve-86dc97cd46-s8twc"] Nov 02 15:49:22.639664 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.639630 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27"] Nov 02 15:49:22.645559 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.645510 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/simulator-w2-fce776-kserve-79b95b659-x6s27"] Nov 02 15:49:22.750049 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.750020 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5"] Nov 02 15:49:22.754114 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:22.754088 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fce776/distinct-w2-fce776-kserve-7bf6bdfc7c-ptlv5"] Nov 02 15:49:23.958776 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:23.958743 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b9b569c-2c19-48e7-bb05-109719649670" path="/var/lib/kubelet/pods/6b9b569c-2c19-48e7-bb05-109719649670/volumes" Nov 02 15:49:23.959206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:23.959190 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" path="/var/lib/kubelet/pods/a8bf04d7-e230-42d5-ad2a-d087d2611533/volumes" Nov 02 15:49:23.959574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:23.959561 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" path="/var/lib/kubelet/pods/fc688f54-c7e1-4244-9a54-c47dc066520a/volumes" Nov 02 15:49:27.936868 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:49:27.936823 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:49:30.340013 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:30.339972 5091 scope.go:122] "RemoveContainer" containerID="7d186786933b7272522da3b70e9fbbfe2e3d189a6efbcff6fa585696a70b4a10" Nov 02 15:49:30.352825 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:30.352793 5091 scope.go:122] "RemoveContainer" containerID="b17a52397afa1c4f6c3789d1e3f4015a0009128d8c0c02eb7f54abb8d6d2c957" Nov 02 15:49:38.722651 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:38.722610 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74"] Nov 02 15:49:38.723186 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:38.722833 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" containerID="cri-o://c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11" gracePeriod=30 Nov 02 15:49:44.023797 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.023760 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:49:44.065963 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.065924 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.065963 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.065970 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.065999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066056 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066085 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066153 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066247 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066182 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.066576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066352 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:44.066576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066469 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:44.066686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066579 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-data\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.066686 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066631 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.066836 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066816 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:44.066888 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.066835 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:44.067043 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.067019 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:49:44.069124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.069087 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 15:49:44.069244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.069201 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:44.069331 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.069311 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167355 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x\") pod \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\" (UID: \"d3b665f2-bce6-4a65-a23e-d092f0e271bc\") " Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167798 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-podinfo\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167824 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-envoy\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167839 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istio-token\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167852 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b665f2-bce6-4a65-a23e-d092f0e271bc-istiod-ca-cert\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167868 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-credential-socket\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.168406 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.167882 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b665f2-bce6-4a65-a23e-d092f0e271bc-workload-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.171679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.171626 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x" pod "d3b665f2-bce6-4a65-a23e-d092f0e271bc" (UID: "d3b665f2-bce6-4a65-a23e-d092f0e271bc"). InnerVolumeSpecName "kube-api-access-gq48x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:49:44.231642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.231598 5091 generic.go:356] "Generic (PLEG): container finished" podID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerID="c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11" exitCode=0 Nov 02 15:49:44.231839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.231658 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" Nov 02 15:49:44.231839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.231688 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" event={"ID":"d3b665f2-bce6-4a65-a23e-d092f0e271bc","Type":"ContainerDied","Data":"c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11"} Nov 02 15:49:44.231839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.231754 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74" event={"ID":"d3b665f2-bce6-4a65-a23e-d092f0e271bc","Type":"ContainerDied","Data":"14351f2bbf5115ab2308e05637892436338e9622bfdbb88a07ae86118566615f"} Nov 02 15:49:44.231839 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.231778 5091 scope.go:122] "RemoveContainer" containerID="c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11" Nov 02 15:49:44.242730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.242706 5091 scope.go:122] "RemoveContainer" containerID="c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11" Nov 02 15:49:44.243079 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:49:44.243055 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11\": container with ID starting with c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11 not found: ID does not exist" containerID="c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11" Nov 02 15:49:44.243157 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.243089 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11"} err="failed to get container status \"c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11\": rpc error: code = NotFound desc = could not find container \"c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11\": container with ID starting with c30c4dff721d0515da65e566d96665c88f322f8b5aff5c1eca48460d59177f11 not found: ID does not exist" Nov 02 15:49:44.269194 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.269008 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq48x\" (UniqueName: \"kubernetes.io/projected/d3b665f2-bce6-4a65-a23e-d092f0e271bc-kube-api-access-gq48x\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:49:44.389074 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.389034 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74"] Nov 02 15:49:44.394761 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:44.394722 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fce776-openshift-default-f6d4dc79c-tql74"] Nov 02 15:49:45.963424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:49:45.963379 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" path="/var/lib/kubelet/pods/d3b665f2-bce6-4a65-a23e-d092f0e271bc/volumes" Nov 02 15:49:57.935459 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:49:57.935417 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:50:27.935598 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:50:27.935556 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:50:30.401089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:50:30.401050 5091 scope.go:122] "RemoveContainer" containerID="2712957ecd1e0a64d3c74706efa8a6e3823f27243d6e196a50540a4184e72333" Nov 02 15:50:57.933475 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:50:57.933388 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:51:27.934334 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:51:27.934285 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:51:37.773060 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773029 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5"] Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773481 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773493 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773506 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773526 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773537 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773543 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773559 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773564 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773580 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1fd161fc-0957-41d4-8acf-0d0be7275149" containerName="revoke-keys" Nov 02 15:51:37.773582 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773585 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd161fc-0957-41d4-8acf-0d0be7275149" containerName="revoke-keys" Nov 02 15:51:37.773924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773655 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b9b569c-2c19-48e7-bb05-109719649670" containerName="main" Nov 02 15:51:37.773924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773662 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bf04d7-e230-42d5-ad2a-d087d2611533" containerName="main" Nov 02 15:51:37.773924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773669 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b665f2-bce6-4a65-a23e-d092f0e271bc" containerName="istio-proxy" Nov 02 15:51:37.773924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773678 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc688f54-c7e1-4244-9a54-c47dc066520a" containerName="main" Nov 02 15:51:37.773924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.773691 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd161fc-0957-41d4-8acf-0d0be7275149" containerName="revoke-keys" Nov 02 15:51:37.776952 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.776929 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.823627 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.823595 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x"] Nov 02 15:51:37.827702 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.827680 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.869844 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.869819 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5"] Nov 02 15:51:37.911033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.910998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911052 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911095 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911128 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzkz\" (UniqueName: \"kubernetes.io/projected/02ebee9d-d52a-4ff9-a0b3-3f071539045d-kube-api-access-vgzkz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911188 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911201 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911207 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911243 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911276 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911323 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911369 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2p5\" (UniqueName: \"kubernetes.io/projected/30c3614d-aa3f-45ed-af95-c4d527cd3f57-kube-api-access-jd2p5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911393 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911414 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911452 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911478 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911497 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:37.911730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.911539 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:37.950882 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:37.950841 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x"] Nov 02 15:51:38.012010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.011975 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2p5\" (UniqueName: \"kubernetes.io/projected/30c3614d-aa3f-45ed-af95-c4d527cd3f57-kube-api-access-jd2p5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012010 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012012 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012034 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012059 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012079 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012100 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012164 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012197 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012303 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012296 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012436 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012549 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzkz\" (UniqueName: \"kubernetes.io/projected/02ebee9d-d52a-4ff9-a0b3-3f071539045d-kube-api-access-vgzkz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012592 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012603 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012660 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012677 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012694 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.012794 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.012764 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013000 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013256 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013475 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013593 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013596 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.013859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.013755 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.015235 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.015207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.015415 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.015396 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.015493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.015425 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.015493 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.015462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.081058 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.080952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2p5\" (UniqueName: \"kubernetes.io/projected/30c3614d-aa3f-45ed-af95-c4d527cd3f57-kube-api-access-jd2p5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.081739 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.081705 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ebee9d-d52a-4ff9-a0b3-3f071539045d-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.081886 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.081848 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzkz\" (UniqueName: \"kubernetes.io/projected/02ebee9d-d52a-4ff9-a0b3-3f071539045d-kube-api-access-vgzkz\") pod \"maas-default-gateway-openshift-default-7f5667f86c-xv6t5\" (UID: \"02ebee9d-d52a-4ff9-a0b3-3f071539045d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.082730 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.082711 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30c3614d-aa3f-45ed-af95-c4d527cd3f57-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-pqg4x\" (UID: \"30c3614d-aa3f-45ed-af95-c4d527cd3f57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.090677 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.090656 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:38.144806 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.144767 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:38.522082 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.522046 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5"] Nov 02 15:51:38.564568 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:51:38.564536 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ebee9d_d52a_4ff9_a0b3_3f071539045d.slice/crio-87ef02e23b00283dc446ff2af5c2d111f1bf3de7f796146e9d77e482bc443c43 WatchSource:0}: Error finding container 87ef02e23b00283dc446ff2af5c2d111f1bf3de7f796146e9d77e482bc443c43: Status 404 returned error can't find the container with id 87ef02e23b00283dc446ff2af5c2d111f1bf3de7f796146e9d77e482bc443c43 Nov 02 15:51:38.566786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.566752 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.566873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.566819 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.566873 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.566851 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.575852 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.575827 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x"] Nov 02 15:51:38.580473 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:51:38.580454 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c3614d_aa3f_45ed_af95_c4d527cd3f57.slice/crio-c92840b0d5658ee9fee51765df51306bd440393b81a33e2f96f01f0c171b5bb0 WatchSource:0}: Error finding container c92840b0d5658ee9fee51765df51306bd440393b81a33e2f96f01f0c171b5bb0: Status 404 returned error can't find the container with id c92840b0d5658ee9fee51765df51306bd440393b81a33e2f96f01f0c171b5bb0 Nov 02 15:51:38.582603 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.582573 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.582706 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.582646 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.582785 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.582731 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 02 15:51:38.886961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.886852 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" event={"ID":"02ebee9d-d52a-4ff9-a0b3-3f071539045d","Type":"ContainerStarted","Data":"3711fd32011677075a8de9817682e0b9eac3cd5c24df0838fdac3cd5e5b81d1a"} Nov 02 15:51:38.886961 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.886921 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" event={"ID":"02ebee9d-d52a-4ff9-a0b3-3f071539045d","Type":"ContainerStarted","Data":"87ef02e23b00283dc446ff2af5c2d111f1bf3de7f796146e9d77e482bc443c43"} Nov 02 15:51:38.889040 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.889010 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" event={"ID":"30c3614d-aa3f-45ed-af95-c4d527cd3f57","Type":"ContainerStarted","Data":"3c470a7534945ad67b3205606b8a514fdd26c3e230a7f58f4c8c39046934dc7c"} Nov 02 15:51:38.889169 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:38.889062 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" event={"ID":"30c3614d-aa3f-45ed-af95-c4d527cd3f57","Type":"ContainerStarted","Data":"c92840b0d5658ee9fee51765df51306bd440393b81a33e2f96f01f0c171b5bb0"} Nov 02 15:51:39.005676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.005623 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podStartSLOduration=2.005607538 podStartE2EDuration="2.005607538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:39.003799018 +0000 UTC m=+3253.168351562" watchObservedRunningTime="2026-11-02 15:51:39.005607538 +0000 UTC m=+3253.170160072" Nov 02 15:51:39.095530 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.095482 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:39.100539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.098804 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:39.100539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.098877 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:39.123678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.123616 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podStartSLOduration=2.123600549 podStartE2EDuration="2.123600549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:51:39.121920823 +0000 UTC m=+3253.286473364" watchObservedRunningTime="2026-11-02 15:51:39.123600549 +0000 UTC m=+3253.288153093" Nov 02 15:51:39.154599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.149573 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:39.154599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.150932 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:39.154599 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:39.150982 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:40.091195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:40.091157 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:40.091614 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:40.091213 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:40.150932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:40.150896 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:40.151205 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:40.151177 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:41.091261 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:41.091222 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:41.091679 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:41.091277 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:41.146742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:41.146705 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:41.147104 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:41.147036 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:42.091055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:42.091012 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:42.091252 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:42.091070 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:42.149399 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:42.149355 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:42.150248 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:42.150045 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:43.091371 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:43.091331 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:43.091611 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:43.091385 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:43.160666 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:43.160624 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:43.161284 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:43.161250 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:44.091863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:44.091826 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:44.092033 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:44.091880 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:44.146542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:44.145983 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:44.146542 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:44.146047 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:45.091063 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:45.091018 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:45.091478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:45.091092 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:45.146023 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:45.145982 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-pqg4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Nov 02 15:51:45.146174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:45.146044 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" podUID="30c3614d-aa3f-45ed-af95-c4d527cd3f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Nov 02 15:51:46.091851 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:46.091807 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-xv6t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" start-of-body= Nov 02 15:51:46.092343 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:46.091877 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" podUID="02ebee9d-d52a-4ff9-a0b3-3f071539045d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.208:15021/healthz/ready\": dial tcp 10.132.0.208:15021: connect: connection refused" Nov 02 15:51:46.904037 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:46.904002 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:46.904424 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:46.904370 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:46.905257 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:46.905235 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-pqg4x" Nov 02 15:51:47.095286 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:47.095256 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:47.095678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:47.095534 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:47.096182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:51:47.096164 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-xv6t5" Nov 02 15:51:57.977285 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:51:57.977240 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:52:27.934018 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:52:27.933977 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:52:28.851804 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:28.851774 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:52:28.870386 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:28.870360 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:52:30.438811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:30.438777 5091 scope.go:122] "RemoveContainer" containerID="be5db6f2f1a3e49d3ffd8003c29becbca1e0e2048b20937e30f4601452c7bae5" Nov 02 15:52:30.449977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:30.449959 5091 scope.go:122] "RemoveContainer" containerID="1c856f96adc4590b7545192bc9989e93129d6d68970b5609a8db24e605236ad4" Nov 02 15:52:30.458239 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:30.458218 5091 scope.go:122] "RemoveContainer" containerID="57efa91190c2de2922e6e3fcbd37abef6ca6e6083bfeeb7e24d7eea5b389dc5c" Nov 02 15:52:45.059742 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:45.059705 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs"] Nov 02 15:52:45.108111 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:45.108077 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6"] Nov 02 15:52:45.111543 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:45.111491 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296cfbd7-td5qs"] Nov 02 15:52:45.114756 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:45.114729 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4396bf6-5c4t6"] Nov 02 15:52:45.974540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:45.972070 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47e07e02-809e-48fd-a468-24cdaf8467b1" path="/var/lib/kubelet/pods/47e07e02-809e-48fd-a468-24cdaf8467b1/volumes" Nov 02 15:52:45.974540 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:45.972707 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f299dba4-aa6b-4dbb-b8eb-bf7475c94831" path="/var/lib/kubelet/pods/f299dba4-aa6b-4dbb-b8eb-bf7475c94831/volumes" Nov 02 15:52:51.046602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:51.046562 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2"] Nov 02 15:52:51.095144 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:51.095110 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11fa493e-jxhf2"] Nov 02 15:52:51.098204 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:51.098175 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw"] Nov 02 15:52:51.099922 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:51.099898 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5820afe5-zxjrw"] Nov 02 15:52:51.960213 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:51.960174 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8216f8b-57f7-4332-9092-e7fef23819d2" path="/var/lib/kubelet/pods/e8216f8b-57f7-4332-9092-e7fef23819d2/volumes" Nov 02 15:52:51.960874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:52:51.960847 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaddab77-172e-4062-b78b-d116ea0b01b5" path="/var/lib/kubelet/pods/eaddab77-172e-4062-b78b-d116ea0b01b5/volumes" Nov 02 15:52:57.934870 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:52:57.934831 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:53:08.054945 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:08.054910 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4"] Nov 02 15:53:08.101676 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:08.101642 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eee8d285-k42x4"] Nov 02 15:53:09.959372 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:09.959335 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff65d5f-a424-4873-bd5e-849e883141de" path="/var/lib/kubelet/pods/7ff65d5f-a424-4873-bd5e-849e883141de/volumes" Nov 02 15:53:27.934290 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:53:27.934242 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:53:30.485498 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:30.485463 5091 scope.go:122] "RemoveContainer" containerID="3ab0f87a15fb2d2baa3c7d9e554ebe376f2735791bdb7a2e2804ab413f1f8f02" Nov 02 15:53:30.496815 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:30.496791 5091 scope.go:122] "RemoveContainer" containerID="9244d67e383d8e2d65f50bee433ab1809a5b478bedba69eb1ee85a306496256a" Nov 02 15:53:30.506152 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:30.506126 5091 scope.go:122] "RemoveContainer" containerID="a813f5ccc42aa5877f69b9635ccf890b24d0b7d43071b2ebd5e537be87f4b59d" Nov 02 15:53:37.040216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:37.039613 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 15:53:37.040216 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:53:37.039714 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 15:53:57.933577 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:53:57.933498 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:54:18.048597 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:18.048562 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x"] Nov 02 15:54:18.095678 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:18.095641 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01b9ad66-q948x"] Nov 02 15:54:19.959055 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:19.959016 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd161fc-0957-41d4-8acf-0d0be7275149" path="/var/lib/kubelet/pods/1fd161fc-0957-41d4-8acf-0d0be7275149/volumes" Nov 02 15:54:27.934252 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:54:27.934205 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:54:30.537985 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.537942 5091 scope.go:122] "RemoveContainer" containerID="f2e06c8e1235f7a852c8818d2797a57f62c4d55f13003fc04c02004ed800c67b" Nov 02 15:54:30.551329 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.551300 5091 scope.go:122] "RemoveContainer" containerID="47eaabfac1ed726e743becccd92e80fd7fc09829e5493c04eda3f5597285edd6" Nov 02 15:54:30.563155 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.563122 5091 scope.go:122] "RemoveContainer" containerID="5b5dfe408e2d2ae33bf500a094934fb01aeef4cc7ab87b3e43f5dfb1b37962e5" Nov 02 15:54:30.575124 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.575012 5091 scope.go:122] "RemoveContainer" containerID="96b4d95c6c3e088911866e90abe31f37efd8e34b4e6149e02544bec1bcdb54af" Nov 02 15:54:30.585182 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.585149 5091 scope.go:122] "RemoveContainer" containerID="e814ed26f8d63c544040bcbdf54727f39e361eb58a90a9340757444bdfffccfb" Nov 02 15:54:30.595934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.595911 5091 scope.go:122] "RemoveContainer" containerID="e3e074428a9f957d1eefdbf463f476589b60248bad8b48b573473711f4b46c1a" Nov 02 15:54:30.604428 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.604346 5091 scope.go:122] "RemoveContainer" containerID="65de3d23533f9d8debc70d49e417087b6411c333f679792773f96294df468de0" Nov 02 15:54:30.614427 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:54:30.614391 5091 scope.go:122] "RemoveContainer" containerID="e21509aa6216c5990d1d16f18d972b4d912becfe4401575ae882b3be5fde9aed" Nov 02 15:54:57.934084 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:54:57.934042 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:55:27.934211 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:55:27.934169 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:55:30.677167 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:55:30.677134 5091 scope.go:122] "RemoveContainer" containerID="15884abd508eb02138746203d34858d1342a660441192f46748004c2f1bfb73b" Nov 02 15:55:57.934449 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:55:57.934402 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:56:27.933696 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:56:27.933654 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:56:57.933194 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:56:57.933156 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:57:05.720304 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.720268 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw"] Nov 02 15:57:05.724287 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.724264 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.784014 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.783982 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.784179 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.784053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.784250 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.784172 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.784318 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.784249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.784375 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.784325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.784423 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.784389 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csc59\" (UniqueName: \"kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.822574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.822542 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a4b243\"/\"default-dockercfg-dx8mn\"" Nov 02 15:57:05.822779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.822591 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a4b243\"/\"openshift-service-ca.crt\"" Nov 02 15:57:05.822779 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.822651 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-a4b243\"/\"kube-root-ca.crt\"" Nov 02 15:57:05.824736 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.824711 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw"] Nov 02 15:57:05.878017 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.877982 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-a4b243\"/\"simulator-single-main-a4b243-kserve-self-signed-certs\"" Nov 02 15:57:05.885642 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.885612 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.885820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.885648 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csc59\" (UniqueName: \"kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.885820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.885681 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.885820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.885738 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.885820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.885769 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.885820 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.885795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.886112 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.886035 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.886193 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.886124 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.886312 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.886294 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.888810 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.888766 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.889127 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.889105 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:05.954862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:05.954830 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csc59\" (UniqueName: \"kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59\") pod \"simulator-single-main-a4b243-kserve-767677c66-l6gjw\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:06.045343 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:06.045253 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:06.431061 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:06.431027 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw"] Nov 02 15:57:06.437578 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:57:06.437550 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a5f669f_4867_414e_b860_e62b3fbec412.slice/crio-912399fdd46ba40b46e0a97bf08cefc4e8ab1164a78b2d1342b9bd73a0fc345d WatchSource:0}: Error finding container 912399fdd46ba40b46e0a97bf08cefc4e8ab1164a78b2d1342b9bd73a0fc345d: Status 404 returned error can't find the container with id 912399fdd46ba40b46e0a97bf08cefc4e8ab1164a78b2d1342b9bd73a0fc345d Nov 02 15:57:06.713715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:06.713630 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" event={"ID":"0a5f669f-4867-414e-b860-e62b3fbec412","Type":"ContainerStarted","Data":"c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf"} Nov 02 15:57:06.713715 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:06.713676 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" event={"ID":"0a5f669f-4867-414e-b860-e62b3fbec412","Type":"ContainerStarted","Data":"912399fdd46ba40b46e0a97bf08cefc4e8ab1164a78b2d1342b9bd73a0fc345d"} Nov 02 15:57:06.826636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:06.826579 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" podStartSLOduration=1.8265648589999999 podStartE2EDuration="1.826564859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:57:06.824879167 +0000 UTC m=+3580.989431719" watchObservedRunningTime="2026-11-02 15:57:06.826564859 +0000 UTC m=+3580.991117399" Nov 02 15:57:07.046262 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:07.046189 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:16.046404 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:16.046364 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:16.058738 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:16.058709 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:16.778183 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:16.778154 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:57:27.934315 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:57:27.934273 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:57:28.883478 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:28.883451 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:57:28.904637 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:28.904604 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 15:57:50.160874 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.160838 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b"] Nov 02 15:57:50.164876 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.164775 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.216419 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.216352 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:57:50.263638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.263609 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:57:50.263638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.263619 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:57:50.263638 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.263639 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:57:50.265902 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.265881 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b"] Nov 02 15:57:50.294109 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.294073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.294292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.294138 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.294292 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.294274 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4gm\" (UniqueName: \"kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.395529 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.395481 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.395721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.395593 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.395721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.395657 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4gm\" (UniqueName: \"kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.396246 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.396219 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.397811 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.397790 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.454602 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.454539 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4gm\" (UniqueName: \"kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm\") pod \"maas-api-revoke-sub-e91560ff-9nm8b\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.478855 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.478829 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:50.846576 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.846465 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b"] Nov 02 15:57:50.852013 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:57:50.851978 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23a01f2_40fe_40a7_a923_04b02024e3d4.slice/crio-a2efcb14f5fa52cb4758a6ab5bbb2a0c86499258b74045e2a1c9594f6dc7fe72 WatchSource:0}: Error finding container a2efcb14f5fa52cb4758a6ab5bbb2a0c86499258b74045e2a1c9594f6dc7fe72: Status 404 returned error can't find the container with id a2efcb14f5fa52cb4758a6ab5bbb2a0c86499258b74045e2a1c9594f6dc7fe72 Nov 02 15:57:50.937979 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:50.937952 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" event={"ID":"c23a01f2-40fe-40a7-a923-04b02024e3d4","Type":"ContainerStarted","Data":"a2efcb14f5fa52cb4758a6ab5bbb2a0c86499258b74045e2a1c9594f6dc7fe72"} Nov 02 15:57:51.945103 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:51.945002 5091 generic.go:356] "Generic (PLEG): container finished" podID="c23a01f2-40fe-40a7-a923-04b02024e3d4" containerID="dc7e0438c74fc7147efa3a295a2b843b8b351c27e7d4a4db5fd13e207b6428e5" exitCode=0 Nov 02 15:57:51.945505 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:51.945086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" event={"ID":"c23a01f2-40fe-40a7-a923-04b02024e3d4","Type":"ContainerDied","Data":"dc7e0438c74fc7147efa3a295a2b843b8b351c27e7d4a4db5fd13e207b6428e5"} Nov 02 15:57:53.087574 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.087546 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:53.223420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.223347 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca\") pod \"c23a01f2-40fe-40a7-a923-04b02024e3d4\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " Nov 02 15:57:53.223420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.223385 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm\" (UniqueName: \"kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm\") pod \"c23a01f2-40fe-40a7-a923-04b02024e3d4\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " Nov 02 15:57:53.223420 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.223419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca\") pod \"c23a01f2-40fe-40a7-a923-04b02024e3d4\" (UID: \"c23a01f2-40fe-40a7-a923-04b02024e3d4\") " Nov 02 15:57:53.223859 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.223821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca" pod "c23a01f2-40fe-40a7-a923-04b02024e3d4" (UID: "c23a01f2-40fe-40a7-a923-04b02024e3d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:57:53.225665 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.225638 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca" pod "c23a01f2-40fe-40a7-a923-04b02024e3d4" (UID: "c23a01f2-40fe-40a7-a923-04b02024e3d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:57:53.225786 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.225719 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm" pod "c23a01f2-40fe-40a7-a923-04b02024e3d4" (UID: "c23a01f2-40fe-40a7-a923-04b02024e3d4"). InnerVolumeSpecName "kube-api-access-2n4gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:57:53.324726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.324698 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:57:53.324726 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.324722 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4gm\" (UniqueName: \"kubernetes.io/projected/c23a01f2-40fe-40a7-a923-04b02024e3d4-kube-api-access-2n4gm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:57:53.324897 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.324733 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c23a01f2-40fe-40a7-a923-04b02024e3d4-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:57:53.956871 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.956840 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" Nov 02 15:57:53.966145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.962067 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b" event={"ID":"c23a01f2-40fe-40a7-a923-04b02024e3d4","Type":"ContainerDied","Data":"a2efcb14f5fa52cb4758a6ab5bbb2a0c86499258b74045e2a1c9594f6dc7fe72"} Nov 02 15:57:53.966145 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:57:53.962136 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2efcb14f5fa52cb4758a6ab5bbb2a0c86499258b74045e2a1c9594f6dc7fe72" Nov 02 15:57:57.934062 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:57:57.934023 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:58:02.311526 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.311491 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx"] Nov 02 15:58:02.311925 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.311912 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c23a01f2-40fe-40a7-a923-04b02024e3d4" containerName="revoke-keys" Nov 02 15:58:02.311975 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.311926 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c23a01f2-40fe-40a7-a923-04b02024e3d4" containerName="revoke-keys" Nov 02 15:58:02.312024 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.312015 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c23a01f2-40fe-40a7-a923-04b02024e3d4" containerName="revoke-keys" Nov 02 15:58:02.315101 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.315084 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.369847 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.369822 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx"] Nov 02 15:58:02.370923 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.370895 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 15:58:02.371046 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.370995 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 15:58:02.408717 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.408690 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.408862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.408734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.408862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.408843 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx7x7\" (UniqueName: \"kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.420362 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.420340 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 15:58:02.420453 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.420406 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 15:58:02.510288 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.510254 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx7x7\" (UniqueName: \"kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.510449 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.510330 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.510449 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.510352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.511081 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.511046 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.512809 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.512784 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.574759 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.574682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx7x7\" (UniqueName: \"kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:02.627924 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:02.627892 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:03.007249 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:58:03.007221 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6e1ecc_0326_438d_8e79_d227cc370f5d.slice/crio-819adc35649e02782c5f7180f6f0d61ce3e3f0a2cf0740f0d2792883a63e0e5f WatchSource:0}: Error finding container 819adc35649e02782c5f7180f6f0d61ce3e3f0a2cf0740f0d2792883a63e0e5f: Status 404 returned error can't find the container with id 819adc35649e02782c5f7180f6f0d61ce3e3f0a2cf0740f0d2792883a63e0e5f Nov 02 15:58:03.007507 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:03.007483 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx"] Nov 02 15:58:04.013195 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:04.013158 5091 generic.go:356] "Generic (PLEG): container finished" podID="3e6e1ecc-0326-438d-8e79-d227cc370f5d" containerID="43dbbae156f8b2dcef28bac3385b16a37b3f719d14b8e0bfd57cdf3c5cde90ad" exitCode=0 Nov 02 15:58:04.013631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:04.013210 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" event={"ID":"3e6e1ecc-0326-438d-8e79-d227cc370f5d","Type":"ContainerDied","Data":"43dbbae156f8b2dcef28bac3385b16a37b3f719d14b8e0bfd57cdf3c5cde90ad"} Nov 02 15:58:04.013631 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:04.013259 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" event={"ID":"3e6e1ecc-0326-438d-8e79-d227cc370f5d","Type":"ContainerStarted","Data":"819adc35649e02782c5f7180f6f0d61ce3e3f0a2cf0740f0d2792883a63e0e5f"} Nov 02 15:58:05.168996 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.168968 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:05.337721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.337626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca\") pod \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " Nov 02 15:58:05.337911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.337717 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca\") pod \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " Nov 02 15:58:05.337911 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.337806 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7\" (UniqueName: \"kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7\") pod \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\" (UID: \"3e6e1ecc-0326-438d-8e79-d227cc370f5d\") " Nov 02 15:58:05.338051 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.338019 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca" pod "3e6e1ecc-0326-438d-8e79-d227cc370f5d" (UID: "3e6e1ecc-0326-438d-8e79-d227cc370f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 15:58:05.338129 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.338106 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:05.339860 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.339827 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca" pod "3e6e1ecc-0326-438d-8e79-d227cc370f5d" (UID: "3e6e1ecc-0326-438d-8e79-d227cc370f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:58:05.339990 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.339906 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7" pod "3e6e1ecc-0326-438d-8e79-d227cc370f5d" (UID: "3e6e1ecc-0326-438d-8e79-d227cc370f5d"). InnerVolumeSpecName "kube-api-access-kx7x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:05.438941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.438910 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx7x7\" (UniqueName: \"kubernetes.io/projected/3e6e1ecc-0326-438d-8e79-d227cc370f5d-kube-api-access-kx7x7\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:05.438941 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:05.438936 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6e1ecc-0326-438d-8e79-d227cc370f5d-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:06.025465 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:06.025434 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" Nov 02 15:58:06.025633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:06.025470 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx" event={"ID":"3e6e1ecc-0326-438d-8e79-d227cc370f5d","Type":"ContainerDied","Data":"819adc35649e02782c5f7180f6f0d61ce3e3f0a2cf0740f0d2792883a63e0e5f"} Nov 02 15:58:06.025633 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:06.025526 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="819adc35649e02782c5f7180f6f0d61ce3e3f0a2cf0740f0d2792883a63e0e5f" Nov 02 15:58:06.738209 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:06.738131 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw"] Nov 02 15:58:20.548021 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:20.547977 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx"] Nov 02 15:58:20.600788 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:20.600754 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a4b243-9dqgx"] Nov 02 15:58:21.758429 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:21.758381 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" podUID="0a5f669f-4867-414e-b860-e62b3fbec412" containerName="main" containerID="cri-o://c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf" gracePeriod=45 Nov 02 15:58:21.767206 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:21.767174 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" podUID="0a5f669f-4867-414e-b860-e62b3fbec412" containerName="main" probeResult="failure" output="Get \"https://10.132.0.210:8000/ready\": dial tcp 10.132.0.210:8000: connect: connection refused" Nov 02 15:58:21.960557 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:21.960498 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6e1ecc-0326-438d-8e79-d227cc370f5d" path="/var/lib/kubelet/pods/3e6e1ecc-0326-438d-8e79-d227cc370f5d/volumes" Nov 02 15:58:22.007468 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.007444 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:58:22.095932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.095860 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache\") pod \"0a5f669f-4867-414e-b860-e62b3fbec412\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " Nov 02 15:58:22.095932 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.095898 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59\" (UniqueName: \"kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59\") pod \"0a5f669f-4867-414e-b860-e62b3fbec412\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " Nov 02 15:58:22.096174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.095937 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm\") pod \"0a5f669f-4867-414e-b860-e62b3fbec412\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " Nov 02 15:58:22.096174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.095969 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home\") pod \"0a5f669f-4867-414e-b860-e62b3fbec412\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " Nov 02 15:58:22.096174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096005 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir\") pod \"0a5f669f-4867-414e-b860-e62b3fbec412\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " Nov 02 15:58:22.096174 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs\") pod \"0a5f669f-4867-414e-b860-e62b3fbec412\" (UID: \"0a5f669f-4867-414e-b860-e62b3fbec412\") " Nov 02 15:58:22.096396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096174 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache" pod "0a5f669f-4867-414e-b860-e62b3fbec412" (UID: "0a5f669f-4867-414e-b860-e62b3fbec412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:22.096396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096222 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home" pod "0a5f669f-4867-414e-b860-e62b3fbec412" (UID: "0a5f669f-4867-414e-b860-e62b3fbec412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:22.096396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096297 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir" pod "0a5f669f-4867-414e-b860-e62b3fbec412" (UID: "0a5f669f-4867-414e-b860-e62b3fbec412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:22.096396 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096392 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:22.096577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096410 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:22.096577 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.096425 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:22.098314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.098287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm" pod "0a5f669f-4867-414e-b860-e62b3fbec412" (UID: "0a5f669f-4867-414e-b860-e62b3fbec412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:22.098474 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.098451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59" pod "0a5f669f-4867-414e-b860-e62b3fbec412" (UID: "0a5f669f-4867-414e-b860-e62b3fbec412"). InnerVolumeSpecName "kube-api-access-csc59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:22.098564 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.098502 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs" pod "0a5f669f-4867-414e-b860-e62b3fbec412" (UID: "0a5f669f-4867-414e-b860-e62b3fbec412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 15:58:22.111862 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.111837 5091 generic.go:356] "Generic (PLEG): container finished" podID="0a5f669f-4867-414e-b860-e62b3fbec412" containerID="c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf" exitCode=0 Nov 02 15:58:22.111977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.111887 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" Nov 02 15:58:22.111977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.111917 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" event={"ID":"0a5f669f-4867-414e-b860-e62b3fbec412","Type":"ContainerDied","Data":"c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf"} Nov 02 15:58:22.111977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.111959 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw" event={"ID":"0a5f669f-4867-414e-b860-e62b3fbec412","Type":"ContainerDied","Data":"912399fdd46ba40b46e0a97bf08cefc4e8ab1164a78b2d1342b9bd73a0fc345d"} Nov 02 15:58:22.111977 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.111977 5091 scope.go:122] "RemoveContainer" containerID="c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf" Nov 02 15:58:22.121314 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.121276 5091 scope.go:122] "RemoveContainer" containerID="c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf" Nov 02 15:58:22.121606 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:58:22.121581 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf\": container with ID starting with c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf not found: ID does not exist" containerID="c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf" Nov 02 15:58:22.121671 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.121608 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf"} err="failed to get container status \"c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf\": rpc error: code = NotFound desc = could not find container \"c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf\": container with ID starting with c380dd1522dd6722cc3757e9d715a74c1975253f5b9e78854c2ad439af088cdf not found: ID does not exist" Nov 02 15:58:22.197556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.197498 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a5f669f-4867-414e-b860-e62b3fbec412-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:22.197556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.197554 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csc59\" (UniqueName: \"kubernetes.io/projected/0a5f669f-4867-414e-b860-e62b3fbec412-kube-api-access-csc59\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:22.197556 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.197568 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a5f669f-4867-414e-b860-e62b3fbec412-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:22.254251 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.254218 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw"] Nov 02 15:58:22.259653 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:22.259626 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-a4b243/simulator-single-main-a4b243-kserve-767677c66-l6gjw"] Nov 02 15:58:23.958934 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:23.958901 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a5f669f-4867-414e-b860-e62b3fbec412" path="/var/lib/kubelet/pods/0a5f669f-4867-414e-b860-e62b3fbec412/volumes" Nov 02 15:58:27.933773 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:58:27.933735 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:58:47.787202 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:47.787166 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv"] Nov 02 15:58:47.787688 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:47.787414 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" podUID="e6c14a88-be07-4d6f-bef8-7802aa502c48" containerName="manager" containerID="cri-o://652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b" gracePeriod=10 Nov 02 15:58:48.058837 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.058757 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:58:48.146636 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.146605 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume\") pod \"e6c14a88-be07-4d6f-bef8-7802aa502c48\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " Nov 02 15:58:48.146817 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.146655 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf\" (UniqueName: \"kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf\") pod \"e6c14a88-be07-4d6f-bef8-7802aa502c48\" (UID: \"e6c14a88-be07-4d6f-bef8-7802aa502c48\") " Nov 02 15:58:48.147089 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.147061 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume" pod "e6c14a88-be07-4d6f-bef8-7802aa502c48" (UID: "e6c14a88-be07-4d6f-bef8-7802aa502c48"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 15:58:48.149042 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.149012 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf" pod "e6c14a88-be07-4d6f-bef8-7802aa502c48" (UID: "e6c14a88-be07-4d6f-bef8-7802aa502c48"). InnerVolumeSpecName "kube-api-access-75hnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 15:58:48.247130 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247095 5091 generic.go:356] "Generic (PLEG): container finished" podID="e6c14a88-be07-4d6f-bef8-7802aa502c48" containerID="652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b" exitCode=0 Nov 02 15:58:48.247309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247146 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" event={"ID":"e6c14a88-be07-4d6f-bef8-7802aa502c48","Type":"ContainerDied","Data":"652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b"} Nov 02 15:58:48.247309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247168 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" Nov 02 15:58:48.247309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv" event={"ID":"e6c14a88-be07-4d6f-bef8-7802aa502c48","Type":"ContainerDied","Data":"896754c2d772d145469408f114eae533e7b8e6148f6a70b787e3bd38d834c34a"} Nov 02 15:58:48.247309 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247246 5091 scope.go:122] "RemoveContainer" containerID="652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b" Nov 02 15:58:48.247539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247384 5091 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e6c14a88-be07-4d6f-bef8-7802aa502c48-extensions-socket-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:48.247539 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.247489 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75hnf\" (UniqueName: \"kubernetes.io/projected/e6c14a88-be07-4d6f-bef8-7802aa502c48-kube-api-access-75hnf\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 15:58:48.261217 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.261184 5091 scope.go:122] "RemoveContainer" containerID="652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b" Nov 02 15:58:48.261526 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:58:48.261484 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b\": container with ID starting with 652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b not found: ID does not exist" containerID="652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b" Nov 02 15:58:48.261617 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.261544 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b"} err="failed to get container status \"652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b\": rpc error: code = NotFound desc = could not find container \"652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b\": container with ID starting with 652ff930b929607e3b6ee8bf0c0a4546f704ed9da30f02ce1050c764a2f22f0b not found: ID does not exist" Nov 02 15:58:48.421282 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.421250 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv"] Nov 02 15:58:48.428030 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:48.428004 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-xr4dv"] Nov 02 15:58:49.958600 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:58:49.958556 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6c14a88-be07-4d6f-bef8-7802aa502c48" path="/var/lib/kubelet/pods/e6c14a88-be07-4d6f-bef8-7802aa502c48/volumes" Nov 02 15:58:57.933263 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:58:57.933226 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:59:27.933418 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:59:27.933378 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 15:59:51.376769 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.376693 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn"] Nov 02 15:59:51.377177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377108 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3e6e1ecc-0326-438d-8e79-d227cc370f5d" containerName="revoke-keys" Nov 02 15:59:51.377177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377121 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6e1ecc-0326-438d-8e79-d227cc370f5d" containerName="revoke-keys" Nov 02 15:59:51.377177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377142 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0a5f669f-4867-414e-b860-e62b3fbec412" containerName="main" Nov 02 15:59:51.377177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377147 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a5f669f-4867-414e-b860-e62b3fbec412" containerName="main" Nov 02 15:59:51.377177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377165 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e6c14a88-be07-4d6f-bef8-7802aa502c48" containerName="manager" Nov 02 15:59:51.377177 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377171 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6c14a88-be07-4d6f-bef8-7802aa502c48" containerName="manager" Nov 02 15:59:51.377381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377231 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6c14a88-be07-4d6f-bef8-7802aa502c48" containerName="manager" Nov 02 15:59:51.377381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377241 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6e1ecc-0326-438d-8e79-d227cc370f5d" containerName="revoke-keys" Nov 02 15:59:51.377381 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.377250 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a5f669f-4867-414e-b860-e62b3fbec412" containerName="main" Nov 02 15:59:51.381382 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.381365 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.416721 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.416686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph6h4\" (UniqueName: \"kubernetes.io/projected/d6d2801c-b13e-4aca-bf65-0ca8807954dc-kube-api-access-ph6h4\") pod \"kuadrant-operator-controller-manager-678d87766f-bhprn\" (UID: \"d6d2801c-b13e-4aca-bf65-0ca8807954dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.416829 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.416753 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d2801c-b13e-4aca-bf65-0ca8807954dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bhprn\" (UID: \"d6d2801c-b13e-4aca-bf65-0ca8807954dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.423004 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.422979 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn"] Nov 02 15:59:51.473798 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.473770 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jnct4\"" Nov 02 15:59:51.517295 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.517264 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph6h4\" (UniqueName: \"kubernetes.io/projected/d6d2801c-b13e-4aca-bf65-0ca8807954dc-kube-api-access-ph6h4\") pod \"kuadrant-operator-controller-manager-678d87766f-bhprn\" (UID: \"d6d2801c-b13e-4aca-bf65-0ca8807954dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.517405 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.517342 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d2801c-b13e-4aca-bf65-0ca8807954dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bhprn\" (UID: \"d6d2801c-b13e-4aca-bf65-0ca8807954dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.517700 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.517682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6d2801c-b13e-4aca-bf65-0ca8807954dc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-bhprn\" (UID: \"d6d2801c-b13e-4aca-bf65-0ca8807954dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.580893 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.580860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph6h4\" (UniqueName: \"kubernetes.io/projected/d6d2801c-b13e-4aca-bf65-0ca8807954dc-kube-api-access-ph6h4\") pod \"kuadrant-operator-controller-manager-678d87766f-bhprn\" (UID: \"d6d2801c-b13e-4aca-bf65-0ca8807954dc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:51.694244 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:51.694170 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:52.240401 ip-10-0-130-229 kubenswrapper[5091]: W1102 15:59:52.240371 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d2801c_b13e_4aca_bf65_0ca8807954dc.slice/crio-4c964f45ea899ff842bf801177c55f3c5ddae32371e4bdd340c2775df8024d28 WatchSource:0}: Error finding container 4c964f45ea899ff842bf801177c55f3c5ddae32371e4bdd340c2775df8024d28: Status 404 returned error can't find the container with id 4c964f45ea899ff842bf801177c55f3c5ddae32371e4bdd340c2775df8024d28 Nov 02 15:59:52.241863 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:52.241832 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn"] Nov 02 15:59:52.568976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:52.568887 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" event={"ID":"d6d2801c-b13e-4aca-bf65-0ca8807954dc","Type":"ContainerStarted","Data":"ba53b1344e4d9c1b7d0718d6767e149e68cc22297f0cb073e714ee2aa889ffb5"} Nov 02 15:59:52.568976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:52.568946 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" event={"ID":"d6d2801c-b13e-4aca-bf65-0ca8807954dc","Type":"ContainerStarted","Data":"4c964f45ea899ff842bf801177c55f3c5ddae32371e4bdd340c2775df8024d28"} Nov 02 15:59:52.568976 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:52.568967 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 15:59:52.698094 ip-10-0-130-229 kubenswrapper[5091]: I1102 15:59:52.698033 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" podStartSLOduration=1.698020424 podStartE2EDuration="1.698020424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 15:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 15:59:52.697757341 +0000 UTC m=+3746.862309882" watchObservedRunningTime="2026-11-02 15:59:52.698020424 +0000 UTC m=+3746.862572964" Nov 02 15:59:57.933380 ip-10-0-130-229 kubenswrapper[5091]: E1102 15:59:57.933344 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:00:03.575651 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:00:03.575620 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-bhprn" Nov 02 16:00:27.933558 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:00:27.933492 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:00:37.044155 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:00:37.043655 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 16:00:37.044155 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:00:37.043734 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 16:00:57.934821 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:00:57.934782 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:01:07.247086 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.247008 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc"] Nov 02 16:01:07.250636 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.250618 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.306114 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.306085 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc"] Nov 02 16:01:07.356068 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.356016 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-61f60231-dockercfg-l4gnt\"" Nov 02 16:01:07.356068 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.356016 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-1d7f44\"" Nov 02 16:01:07.389693 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.389663 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcnb\" (UniqueName: \"kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb\") pod \"payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.389861 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.389700 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.491100 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.491066 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcnb\" (UniqueName: \"kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb\") pod \"payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.491267 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.491105 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.491884 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.491863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.557179 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.557106 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcnb\" (UniqueName: \"kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb\") pod \"payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.563703 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.563681 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:07.936648 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.936615 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc"] Nov 02 16:01:07.937244 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:01:07.937218 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9642f3_3d5c_4ff8_ba90_20991c797342.slice/crio-04c1ee88e2ddeba921250a11582d28f15545c90e2520964bb21d2519fa68c2aa WatchSource:0}: Error finding container 04c1ee88e2ddeba921250a11582d28f15545c90e2520964bb21d2519fa68c2aa: Status 404 returned error can't find the container with id 04c1ee88e2ddeba921250a11582d28f15545c90e2520964bb21d2519fa68c2aa Nov 02 16:01:07.972844 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:07.972811 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" event={"ID":"ae9642f3-3d5c-4ff8-ba90-20991c797342","Type":"ContainerStarted","Data":"04c1ee88e2ddeba921250a11582d28f15545c90e2520964bb21d2519fa68c2aa"} Nov 02 16:01:08.979145 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:08.979111 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" event={"ID":"ae9642f3-3d5c-4ff8-ba90-20991c797342","Type":"ContainerStarted","Data":"cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5"} Nov 02 16:01:08.979648 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:08.979259 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:09.102865 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:09.102818 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" podStartSLOduration=2.102805229 podStartE2EDuration="2.102805229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:09.100855976 +0000 UTC m=+3823.265408516" watchObservedRunningTime="2026-11-02 16:01:09.102805229 +0000 UTC m=+3823.267357768" Nov 02 16:01:19.986157 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:19.986130 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:01:27.934341 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:01:27.934300 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:01:32.059746 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.059712 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48"] Nov 02 16:01:32.063215 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.063197 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.089622 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.089586 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.089797 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.089671 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.089797 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.089711 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.089797 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.089727 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2b52\" (UniqueName: \"kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.089797 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.089744 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.089971 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.089892 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.121585 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.121548 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48"] Nov 02 16:01:32.171280 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.171248 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1d7f44\"/\"openshift-service-ca.crt\"" Nov 02 16:01:32.171548 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.171498 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-1d7f44\"/\"kube-root-ca.crt\"" Nov 02 16:01:32.173052 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.173034 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1d7f44\"/\"default-dockercfg-9hdw4\"" Nov 02 16:01:32.173281 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.173265 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-1d7f44\"/\"simulator-single-main-1d7f44-kserve-self-signed-certs\"" Nov 02 16:01:32.191082 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.191055 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.191391 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.191358 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.191596 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.191569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.191713 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.191621 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.191713 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.191653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2b52\" (UniqueName: \"kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.191713 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.191683 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.192409 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.192386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.192661 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.192637 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.192795 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.192773 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.196663 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.196635 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.201311 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.201286 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.287478 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.287443 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2b52\" (UniqueName: \"kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52\") pod \"simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.386103 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.386064 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:32.773781 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:32.773632 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48"] Nov 02 16:01:33.112105 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:33.112018 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" event={"ID":"799dfcba-44b0-40f1-9e84-c2718a1f85cc","Type":"ContainerStarted","Data":"529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a"} Nov 02 16:01:33.112105 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:33.112063 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" event={"ID":"799dfcba-44b0-40f1-9e84-c2718a1f85cc","Type":"ContainerStarted","Data":"ffaedc5d3ccccf96ebd53e317428312c91f56393d36cb8ea987d2106acf279ff"} Nov 02 16:01:33.240627 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:33.240577 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" podStartSLOduration=1.240563417 podStartE2EDuration="1.240563417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:01:33.239446732 +0000 UTC m=+3847.403999272" watchObservedRunningTime="2026-11-02 16:01:33.240563417 +0000 UTC m=+3847.405115956" Nov 02 16:01:33.386786 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:33.386754 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:42.386699 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:42.386664 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:42.399380 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:42.399356 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:43.170799 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:01:43.170769 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:01:57.933740 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:01:57.933690 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:02:17.025577 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.025544 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s"] Nov 02 16:02:17.029289 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.029268 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.082225 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.082186 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:02:17.082600 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.082575 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:02:17.111707 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.111682 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.111828 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.111763 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.111903 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.111834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d5h4\" (UniqueName: \"kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.129072 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.129035 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:02:17.129211 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.129196 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:02:17.131553 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.131534 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s"] Nov 02 16:02:17.212878 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.212853 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.212985 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.212912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.212985 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.212964 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d5h4\" (UniqueName: \"kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.213684 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.213656 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.215225 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.215203 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.271811 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.271791 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d5h4\" (UniqueName: \"kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4\") pod \"maas-api-revoke-sub-feb1362e-bw48s\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.342436 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.342365 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:17.711788 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:17.711753 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s"] Nov 02 16:02:17.717407 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:02:17.717375 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e4b2d1_4db6_43fa_b02e_894571459898.slice/crio-f40b652f40427912b5e939ee5e603f8ecdb53c1a34e10b3f59a0843976df65a7 WatchSource:0}: Error finding container f40b652f40427912b5e939ee5e603f8ecdb53c1a34e10b3f59a0843976df65a7: Status 404 returned error can't find the container with id f40b652f40427912b5e939ee5e603f8ecdb53c1a34e10b3f59a0843976df65a7 Nov 02 16:02:18.340589 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:18.340496 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" event={"ID":"f3e4b2d1-4db6-43fa-b02e-894571459898","Type":"ContainerStarted","Data":"216d32d0ed19adac640c7b9f477e40dd89fa38842b498bae47b2dbe2931bdc53"} Nov 02 16:02:18.340589 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:18.340564 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" event={"ID":"f3e4b2d1-4db6-43fa-b02e-894571459898","Type":"ContainerStarted","Data":"f40b652f40427912b5e939ee5e603f8ecdb53c1a34e10b3f59a0843976df65a7"} Nov 02 16:02:18.453483 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:18.453432 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" podStartSLOduration=2.453418487 podStartE2EDuration="2.453418487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:02:18.451007161 +0000 UTC m=+3892.615559702" watchObservedRunningTime="2026-11-02 16:02:18.453418487 +0000 UTC m=+3892.617971026" Nov 02 16:02:19.347139 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:19.347106 5091 generic.go:356] "Generic (PLEG): container finished" podID="f3e4b2d1-4db6-43fa-b02e-894571459898" containerID="216d32d0ed19adac640c7b9f477e40dd89fa38842b498bae47b2dbe2931bdc53" exitCode=0 Nov 02 16:02:19.347576 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:19.347191 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" event={"ID":"f3e4b2d1-4db6-43fa-b02e-894571459898","Type":"ContainerDied","Data":"216d32d0ed19adac640c7b9f477e40dd89fa38842b498bae47b2dbe2931bdc53"} Nov 02 16:02:20.494259 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.494233 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:20.543815 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.543791 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca\") pod \"f3e4b2d1-4db6-43fa-b02e-894571459898\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " Nov 02 16:02:20.543960 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.543862 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4\" (UniqueName: \"kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4\") pod \"f3e4b2d1-4db6-43fa-b02e-894571459898\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " Nov 02 16:02:20.544010 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.543962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca\") pod \"f3e4b2d1-4db6-43fa-b02e-894571459898\" (UID: \"f3e4b2d1-4db6-43fa-b02e-894571459898\") " Nov 02 16:02:20.544309 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.544285 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca" pod "f3e4b2d1-4db6-43fa-b02e-894571459898" (UID: "f3e4b2d1-4db6-43fa-b02e-894571459898"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:02:20.545795 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.545775 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca" pod "f3e4b2d1-4db6-43fa-b02e-894571459898" (UID: "f3e4b2d1-4db6-43fa-b02e-894571459898"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:02:20.546002 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.545987 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4" pod "f3e4b2d1-4db6-43fa-b02e-894571459898" (UID: "f3e4b2d1-4db6-43fa-b02e-894571459898"). InnerVolumeSpecName "kube-api-access-2d5h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:02:20.645187 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.645159 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:20.645187 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.645187 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e4b2d1-4db6-43fa-b02e-894571459898-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:20.645378 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:20.645197 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d5h4\" (UniqueName: \"kubernetes.io/projected/f3e4b2d1-4db6-43fa-b02e-894571459898-kube-api-access-2d5h4\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:21.358285 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:21.358255 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" Nov 02 16:02:21.358285 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:21.358264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s" event={"ID":"f3e4b2d1-4db6-43fa-b02e-894571459898","Type":"ContainerDied","Data":"f40b652f40427912b5e939ee5e603f8ecdb53c1a34e10b3f59a0843976df65a7"} Nov 02 16:02:21.358504 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:21.358306 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f40b652f40427912b5e939ee5e603f8ecdb53c1a34e10b3f59a0843976df65a7" Nov 02 16:02:27.933026 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:02:27.932981 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:02:28.735536 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.735486 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5"] Nov 02 16:02:28.736005 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.735985 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f3e4b2d1-4db6-43fa-b02e-894571459898" containerName="revoke-keys" Nov 02 16:02:28.736103 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.736008 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e4b2d1-4db6-43fa-b02e-894571459898" containerName="revoke-keys" Nov 02 16:02:28.736165 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.736147 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e4b2d1-4db6-43fa-b02e-894571459898" containerName="revoke-keys" Nov 02 16:02:28.739548 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.739512 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.792456 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.792424 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:02:28.822252 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.822226 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.822432 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.822403 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.822507 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.822493 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2rdk\" (UniqueName: \"kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.839556 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.839510 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ffvrz\"" Nov 02 16:02:28.839784 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.839764 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:02:28.839869 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.839837 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:02:28.841940 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.841919 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5"] Nov 02 16:02:28.918193 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.918170 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 16:02:28.922952 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.922926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2rdk\" (UniqueName: \"kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.923052 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.922990 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.923114 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.923075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.923785 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.923760 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.925802 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.925781 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:28.941760 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.941739 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 16:02:28.982911 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:28.982880 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2rdk\" (UniqueName: \"kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk\") pod \"maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:29.061310 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:29.061253 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:29.431031 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:29.430995 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5"] Nov 02 16:02:29.435563 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:02:29.435531 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244f405e_1ece_47eb_9bc3_af8c56348e6f.slice/crio-90aa01c9114654b7e18531802e5444dada908dd10092e36dedeade5f9540d8d9 WatchSource:0}: Error finding container 90aa01c9114654b7e18531802e5444dada908dd10092e36dedeade5f9540d8d9: Status 404 returned error can't find the container with id 90aa01c9114654b7e18531802e5444dada908dd10092e36dedeade5f9540d8d9 Nov 02 16:02:30.404944 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:30.404914 5091 generic.go:356] "Generic (PLEG): container finished" podID="244f405e-1ece-47eb-9bc3-af8c56348e6f" containerID="1b1a2c2b5f90af89c568047472b4795b6096f01386cd1373a8879aa2ed0e2beb" exitCode=0 Nov 02 16:02:30.405358 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:30.405005 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" event={"ID":"244f405e-1ece-47eb-9bc3-af8c56348e6f","Type":"ContainerDied","Data":"1b1a2c2b5f90af89c568047472b4795b6096f01386cd1373a8879aa2ed0e2beb"} Nov 02 16:02:30.405358 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:30.405072 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" event={"ID":"244f405e-1ece-47eb-9bc3-af8c56348e6f","Type":"ContainerStarted","Data":"90aa01c9114654b7e18531802e5444dada908dd10092e36dedeade5f9540d8d9"} Nov 02 16:02:31.560983 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.560951 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:31.650412 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.650381 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk\" (UniqueName: \"kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk\") pod \"244f405e-1ece-47eb-9bc3-af8c56348e6f\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " Nov 02 16:02:31.650597 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.650437 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca\") pod \"244f405e-1ece-47eb-9bc3-af8c56348e6f\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " Nov 02 16:02:31.650597 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.650494 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca\") pod \"244f405e-1ece-47eb-9bc3-af8c56348e6f\" (UID: \"244f405e-1ece-47eb-9bc3-af8c56348e6f\") " Nov 02 16:02:31.650929 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.650902 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca" pod "244f405e-1ece-47eb-9bc3-af8c56348e6f" (UID: "244f405e-1ece-47eb-9bc3-af8c56348e6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:02:31.652497 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.652471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk" pod "244f405e-1ece-47eb-9bc3-af8c56348e6f" (UID: "244f405e-1ece-47eb-9bc3-af8c56348e6f"). InnerVolumeSpecName "kube-api-access-p2rdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:02:31.652732 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.652704 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca" pod "244f405e-1ece-47eb-9bc3-af8c56348e6f" (UID: "244f405e-1ece-47eb-9bc3-af8c56348e6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:02:31.751507 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.751411 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2rdk\" (UniqueName: \"kubernetes.io/projected/244f405e-1ece-47eb-9bc3-af8c56348e6f-kube-api-access-p2rdk\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:31.751507 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.751448 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:31.751507 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:31.751463 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f405e-1ece-47eb-9bc3-af8c56348e6f-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:32.417735 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:32.417709 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" Nov 02 16:02:32.417906 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:32.417708 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5" event={"ID":"244f405e-1ece-47eb-9bc3-af8c56348e6f","Type":"ContainerDied","Data":"90aa01c9114654b7e18531802e5444dada908dd10092e36dedeade5f9540d8d9"} Nov 02 16:02:32.417906 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:32.417834 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90aa01c9114654b7e18531802e5444dada908dd10092e36dedeade5f9540d8d9" Nov 02 16:02:33.168120 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:33.168081 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48"] Nov 02 16:02:40.277895 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:40.277810 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc"] Nov 02 16:02:45.278695 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:45.278655 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerName="payload-pre-processing" containerID="cri-o://cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5" gracePeriod=25 Nov 02 16:02:45.588255 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:45.588174 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5"] Nov 02 16:02:45.634973 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:45.634945 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-1d7f44-h5pv5"] Nov 02 16:02:45.959802 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:45.959767 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244f405e-1ece-47eb-9bc3-af8c56348e6f" path="/var/lib/kubelet/pods/244f405e-1ece-47eb-9bc3-af8c56348e6f/volumes" Nov 02 16:02:48.188353 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.188293 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" podUID="799dfcba-44b0-40f1-9e84-c2718a1f85cc" containerName="main" containerID="cri-o://529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a" gracePeriod=45 Nov 02 16:02:48.440913 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.440857 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:02:48.504942 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.504910 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs\") pod \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " Nov 02 16:02:48.505117 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.504968 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir\") pod \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " Nov 02 16:02:48.505117 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505002 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home\") pod \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " Nov 02 16:02:48.505117 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505056 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm\") pod \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " Nov 02 16:02:48.505117 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505092 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52\" (UniqueName: \"kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52\") pod \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " Nov 02 16:02:48.505309 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505168 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache\") pod \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\" (UID: \"799dfcba-44b0-40f1-9e84-c2718a1f85cc\") " Nov 02 16:02:48.505309 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505216 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir" pod "799dfcba-44b0-40f1-9e84-c2718a1f85cc" (UID: "799dfcba-44b0-40f1-9e84-c2718a1f85cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:48.505387 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505339 5091 generic.go:356] "Generic (PLEG): container finished" podID="799dfcba-44b0-40f1-9e84-c2718a1f85cc" containerID="529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a" exitCode=0 Nov 02 16:02:48.505387 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505381 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" Nov 02 16:02:48.505454 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505415 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:48.505489 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" event={"ID":"799dfcba-44b0-40f1-9e84-c2718a1f85cc","Type":"ContainerDied","Data":"529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a"} Nov 02 16:02:48.505550 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48" event={"ID":"799dfcba-44b0-40f1-9e84-c2718a1f85cc","Type":"ContainerDied","Data":"ffaedc5d3ccccf96ebd53e317428312c91f56393d36cb8ea987d2106acf279ff"} Nov 02 16:02:48.505550 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505542 5091 scope.go:122] "RemoveContainer" containerID="529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a" Nov 02 16:02:48.505695 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505675 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home" pod "799dfcba-44b0-40f1-9e84-c2718a1f85cc" (UID: "799dfcba-44b0-40f1-9e84-c2718a1f85cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:48.505818 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.505789 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache" pod "799dfcba-44b0-40f1-9e84-c2718a1f85cc" (UID: "799dfcba-44b0-40f1-9e84-c2718a1f85cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:48.507362 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.507340 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs" pod "799dfcba-44b0-40f1-9e84-c2718a1f85cc" (UID: "799dfcba-44b0-40f1-9e84-c2718a1f85cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:02:48.507633 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.507605 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm" pod "799dfcba-44b0-40f1-9e84-c2718a1f85cc" (UID: "799dfcba-44b0-40f1-9e84-c2718a1f85cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:02:48.507877 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.507857 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52" pod "799dfcba-44b0-40f1-9e84-c2718a1f85cc" (UID: "799dfcba-44b0-40f1-9e84-c2718a1f85cc"). InnerVolumeSpecName "kube-api-access-x2b52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:02:48.526287 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.526270 5091 scope.go:122] "RemoveContainer" containerID="529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a" Nov 02 16:02:48.526561 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:02:48.526540 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a\": container with ID starting with 529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a not found: ID does not exist" containerID="529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a" Nov 02 16:02:48.526625 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.526571 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a"} err="failed to get container status \"529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a\": rpc error: code = NotFound desc = could not find container \"529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a\": container with ID starting with 529628f4687549031711b05bd046ad472639db5694d5bda8837845affbaed58a not found: ID does not exist" Nov 02 16:02:48.605897 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.605865 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:48.605897 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.605897 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2b52\" (UniqueName: \"kubernetes.io/projected/799dfcba-44b0-40f1-9e84-c2718a1f85cc-kube-api-access-x2b52\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:48.606137 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.605910 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:48.606137 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.605919 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799dfcba-44b0-40f1-9e84-c2718a1f85cc-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:48.606137 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.605928 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/799dfcba-44b0-40f1-9e84-c2718a1f85cc-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:02:48.920962 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.920930 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48"] Nov 02 16:02:48.925772 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:48.925745 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-1d7f44/simulator-single-main-1d7f44-kserve-58866d6cc8-wnx48"] Nov 02 16:02:49.959204 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:49.959162 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="799dfcba-44b0-40f1-9e84-c2718a1f85cc" path="/var/lib/kubelet/pods/799dfcba-44b0-40f1-9e84-c2718a1f85cc/volumes" Nov 02 16:02:49.985275 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:49.985238 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 16:02:54.046066 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:54.046031 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b"] Nov 02 16:02:54.093469 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:54.093435 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e91560ff-9nm8b"] Nov 02 16:02:55.958278 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:55.958242 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c23a01f2-40fe-40a7-a923-04b02024e3d4" path="/var/lib/kubelet/pods/c23a01f2-40fe-40a7-a923-04b02024e3d4/volumes" Nov 02 16:02:57.933657 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:02:57.933613 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:02:59.985222 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:02:59.985186 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 16:03:05.529825 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.529802 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:03:05.594186 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.594122 5091 generic.go:356] "Generic (PLEG): container finished" podID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerID="cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5" exitCode=0 Nov 02 16:03:05.594186 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.594165 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" Nov 02 16:03:05.594186 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.594179 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" event={"ID":"ae9642f3-3d5c-4ff8-ba90-20991c797342","Type":"ContainerDied","Data":"cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5"} Nov 02 16:03:05.594397 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.594230 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc" event={"ID":"ae9642f3-3d5c-4ff8-ba90-20991c797342","Type":"ContainerDied","Data":"04c1ee88e2ddeba921250a11582d28f15545c90e2520964bb21d2519fa68c2aa"} Nov 02 16:03:05.594397 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.594253 5091 scope.go:122] "RemoveContainer" containerID="cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5" Nov 02 16:03:05.602916 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.602896 5091 scope.go:122] "RemoveContainer" containerID="cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5" Nov 02 16:03:05.603190 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:03:05.603168 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5\": container with ID starting with cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5 not found: ID does not exist" containerID="cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5" Nov 02 16:03:05.603245 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.603199 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5"} err="failed to get container status \"cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5\": rpc error: code = NotFound desc = could not find container \"cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5\": container with ID starting with cbc7ffbefef1e32d0ce5021cd90a4f93cb0a292ee764d6257e6632b57b8001b5 not found: ID does not exist" Nov 02 16:03:05.666464 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.666440 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb\" (UniqueName: \"kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb\") pod \"ae9642f3-3d5c-4ff8-ba90-20991c797342\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " Nov 02 16:03:05.666618 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.666557 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume\") pod \"ae9642f3-3d5c-4ff8-ba90-20991c797342\" (UID: \"ae9642f3-3d5c-4ff8-ba90-20991c797342\") " Nov 02 16:03:05.666928 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.666901 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume" pod "ae9642f3-3d5c-4ff8-ba90-20991c797342" (UID: "ae9642f3-3d5c-4ff8-ba90-20991c797342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:03:05.668506 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.668480 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb" pod "ae9642f3-3d5c-4ff8-ba90-20991c797342" (UID: "ae9642f3-3d5c-4ff8-ba90-20991c797342"). InnerVolumeSpecName "kube-api-access-2mcnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:03:05.767564 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.767511 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mcnb\" (UniqueName: \"kubernetes.io/projected/ae9642f3-3d5c-4ff8-ba90-20991c797342-kube-api-access-2mcnb\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:03:05.767564 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:05.767561 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae9642f3-3d5c-4ff8-ba90-20991c797342-plugins-config-volume\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:03:06.015362 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:06.015326 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc"] Nov 02 16:03:06.015934 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:06.015905 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-1d7f44-79f4cpdbwc"] Nov 02 16:03:07.958693 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:07.958658 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" path="/var/lib/kubelet/pods/ae9642f3-3d5c-4ff8-ba90-20991c797342/volumes" Nov 02 16:03:12.290629 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.290593 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr"] Nov 02 16:03:12.291025 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.290985 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerName="payload-pre-processing" Nov 02 16:03:12.291025 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.290996 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerName="payload-pre-processing" Nov 02 16:03:12.291104 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291029 5091 cpu_manager.go:414] "containerMap: removing container" podUID="799dfcba-44b0-40f1-9e84-c2718a1f85cc" containerName="main" Nov 02 16:03:12.291104 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291035 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="799dfcba-44b0-40f1-9e84-c2718a1f85cc" containerName="main" Nov 02 16:03:12.291104 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291045 5091 cpu_manager.go:414] "containerMap: removing container" podUID="244f405e-1ece-47eb-9bc3-af8c56348e6f" containerName="revoke-keys" Nov 02 16:03:12.291104 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291050 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244f405e-1ece-47eb-9bc3-af8c56348e6f" containerName="revoke-keys" Nov 02 16:03:12.291277 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291106 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="799dfcba-44b0-40f1-9e84-c2718a1f85cc" containerName="main" Nov 02 16:03:12.291277 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291116 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="244f405e-1ece-47eb-9bc3-af8c56348e6f" containerName="revoke-keys" Nov 02 16:03:12.291277 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.291124 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae9642f3-3d5c-4ff8-ba90-20991c797342" containerName="payload-pre-processing" Nov 02 16:03:12.295683 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.295656 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.394411 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.394380 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-49zzg\"" Nov 02 16:03:12.394411 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.394377 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d5201e83-kserve-self-signed-certs\"" Nov 02 16:03:12.394411 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.394418 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 16:03:12.394718 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.394486 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 16:03:12.399127 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.399098 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr"] Nov 02 16:03:12.427118 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.427083 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.427290 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.427129 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.427290 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.427264 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.427381 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.427358 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.427443 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.427425 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.427536 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.427510 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xb9\" (UniqueName: \"kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.528599 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.528562 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.528781 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.528625 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xb9\" (UniqueName: \"kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.528781 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.528648 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.528781 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.528675 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.528781 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.528703 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.528781 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.528743 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.529086 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.529061 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.529128 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.529107 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.529191 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.529160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.530951 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.530932 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.531294 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.531277 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.587045 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.586979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xb9\" (UniqueName: \"kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9\") pod \"e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.615077 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.615043 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:12.996255 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:12.996219 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr"] Nov 02 16:03:13.001989 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:03:13.001961 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2868b942_06be_4bad_9fa2_83db11e7c5eb.slice/crio-1a695a31d826ebf4ff0fb32db9d28048ea15e8a72c90e32f3d3ba51b82a74f83 WatchSource:0}: Error finding container 1a695a31d826ebf4ff0fb32db9d28048ea15e8a72c90e32f3d3ba51b82a74f83: Status 404 returned error can't find the container with id 1a695a31d826ebf4ff0fb32db9d28048ea15e8a72c90e32f3d3ba51b82a74f83 Nov 02 16:03:13.638936 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:13.638900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" event={"ID":"2868b942-06be-4bad-9fa2-83db11e7c5eb","Type":"ContainerStarted","Data":"a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096"} Nov 02 16:03:13.638936 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:13.638941 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" event={"ID":"2868b942-06be-4bad-9fa2-83db11e7c5eb","Type":"ContainerStarted","Data":"1a695a31d826ebf4ff0fb32db9d28048ea15e8a72c90e32f3d3ba51b82a74f83"} Nov 02 16:03:13.751962 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:13.751912 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" podStartSLOduration=1.7518995990000001 podStartE2EDuration="1.751899599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:03:13.750632255 +0000 UTC m=+3947.915184794" watchObservedRunningTime="2026-11-02 16:03:13.751899599 +0000 UTC m=+3947.916452139" Nov 02 16:03:14.615509 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:14.615473 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:22.615675 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:22.615640 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:22.628143 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:22.628117 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:22.709621 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:03:22.709591 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:03:27.933350 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:03:27.933308 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:03:57.932913 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:03:57.932872 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:04:27.933757 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:04:27.933715 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:04:28.579019 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:28.578983 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr"] Nov 02 16:04:30.780251 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:30.780217 5091 scope.go:122] "RemoveContainer" containerID="43dbbae156f8b2dcef28bac3385b16a37b3f719d14b8e0bfd57cdf3c5cde90ad" Nov 02 16:04:30.789260 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:30.789238 5091 scope.go:122] "RemoveContainer" containerID="dc7e0438c74fc7147efa3a295a2b843b8b351c27e7d4a4db5fd13e207b6428e5" Nov 02 16:04:43.599088 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.599022 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" podUID="2868b942-06be-4bad-9fa2-83db11e7c5eb" containerName="main" containerID="cri-o://a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096" gracePeriod=45 Nov 02 16:04:43.699384 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.699338 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" podUID="2868b942-06be-4bad-9fa2-83db11e7c5eb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.218:8000/ready\": dial tcp 10.132.0.218:8000: connect: connection refused" Nov 02 16:04:43.859779 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.859723 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:04:43.927216 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927185 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9\" (UniqueName: \"kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9\") pod \"2868b942-06be-4bad-9fa2-83db11e7c5eb\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " Nov 02 16:04:43.927395 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927227 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir\") pod \"2868b942-06be-4bad-9fa2-83db11e7c5eb\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " Nov 02 16:04:43.927395 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs\" (UniqueName: \"kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs\") pod \"2868b942-06be-4bad-9fa2-83db11e7c5eb\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " Nov 02 16:04:43.927395 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927328 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm\") pod \"2868b942-06be-4bad-9fa2-83db11e7c5eb\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " Nov 02 16:04:43.927668 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927450 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home\") pod \"2868b942-06be-4bad-9fa2-83db11e7c5eb\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " Nov 02 16:04:43.927668 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927480 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache\") pod \"2868b942-06be-4bad-9fa2-83db11e7c5eb\" (UID: \"2868b942-06be-4bad-9fa2-83db11e7c5eb\") " Nov 02 16:04:43.927668 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927562 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir" pod "2868b942-06be-4bad-9fa2-83db11e7c5eb" (UID: "2868b942-06be-4bad-9fa2-83db11e7c5eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:43.927840 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927692 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home" pod "2868b942-06be-4bad-9fa2-83db11e7c5eb" (UID: "2868b942-06be-4bad-9fa2-83db11e7c5eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:43.927840 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927829 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache" pod "2868b942-06be-4bad-9fa2-83db11e7c5eb" (UID: "2868b942-06be-4bad-9fa2-83db11e7c5eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:43.927937 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927845 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-home\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:43.927937 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.927864 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-tmp-dir\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:43.929641 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.929619 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs" pod "2868b942-06be-4bad-9fa2-83db11e7c5eb" (UID: "2868b942-06be-4bad-9fa2-83db11e7c5eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:04:43.929732 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.929647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9" pod "2868b942-06be-4bad-9fa2-83db11e7c5eb" (UID: "2868b942-06be-4bad-9fa2-83db11e7c5eb"). InnerVolumeSpecName "kube-api-access-c9xb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:04:43.929732 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:43.929677 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm" pod "2868b942-06be-4bad-9fa2-83db11e7c5eb" (UID: "2868b942-06be-4bad-9fa2-83db11e7c5eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 16:04:44.029089 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.029057 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-dshm\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:44.029089 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.029086 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2868b942-06be-4bad-9fa2-83db11e7c5eb-model-cache\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:44.029286 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.029098 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9xb9\" (UniqueName: \"kubernetes.io/projected/2868b942-06be-4bad-9fa2-83db11e7c5eb-kube-api-access-c9xb9\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:44.029286 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.029108 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2868b942-06be-4bad-9fa2-83db11e7c5eb-tls-certs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:04:44.188239 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.188203 5091 generic.go:356] "Generic (PLEG): container finished" podID="2868b942-06be-4bad-9fa2-83db11e7c5eb" containerID="a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096" exitCode=0 Nov 02 16:04:44.188448 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.188243 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" Nov 02 16:04:44.188448 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.188291 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" event={"ID":"2868b942-06be-4bad-9fa2-83db11e7c5eb","Type":"ContainerDied","Data":"a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096"} Nov 02 16:04:44.188448 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.188338 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr" event={"ID":"2868b942-06be-4bad-9fa2-83db11e7c5eb","Type":"ContainerDied","Data":"1a695a31d826ebf4ff0fb32db9d28048ea15e8a72c90e32f3d3ba51b82a74f83"} Nov 02 16:04:44.188448 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.188360 5091 scope.go:122] "RemoveContainer" containerID="a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096" Nov 02 16:04:44.197271 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.197254 5091 scope.go:122] "RemoveContainer" containerID="a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096" Nov 02 16:04:44.197548 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:04:44.197509 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096\": container with ID starting with a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096 not found: ID does not exist" containerID="a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096" Nov 02 16:04:44.197641 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.197555 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096"} err="failed to get container status \"a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096\": rpc error: code = NotFound desc = could not find container \"a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096\": container with ID starting with a7276958cbd3c07274f84343e5e3aa7efb71132a5c865d4c89e4bc27057db096 not found: ID does not exist" Nov 02 16:04:44.344690 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.344661 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr"] Nov 02 16:04:44.349119 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:44.349098 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d5201e83-kserve-64dfdc95d5-j4jbr"] Nov 02 16:04:45.959998 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:45.959964 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2868b942-06be-4bad-9fa2-83db11e7c5eb" path="/var/lib/kubelet/pods/2868b942-06be-4bad-9fa2-83db11e7c5eb/volumes" Nov 02 16:04:52.048677 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:52.048629 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r2p85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 16:04:52.049300 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:04:52.049261 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r2p85" podUID="ee194769-7328-456a-b24d-c368212d5672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 16:04:57.933401 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:04:57.933360 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:05:27.935293 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:05:27.935245 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:05:57.934324 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:05:57.934243 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:06:02.434865 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.434830 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7"] Nov 02 16:06:02.435378 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.435257 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2868b942-06be-4bad-9fa2-83db11e7c5eb" containerName="main" Nov 02 16:06:02.435378 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.435269 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2868b942-06be-4bad-9fa2-83db11e7c5eb" containerName="main" Nov 02 16:06:02.435378 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.435357 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2868b942-06be-4bad-9fa2-83db11e7c5eb" containerName="main" Nov 02 16:06:02.438056 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.438040 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.502539 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.502480 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.502705 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.502564 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.502705 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.502652 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2h5z\" (UniqueName: \"kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.535000 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.534967 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:06:02.535000 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.534989 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j2bsg\"" Nov 02 16:06:02.535000 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.534995 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:06:02.535000 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.534999 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:06:02.537176 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.537154 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7"] Nov 02 16:06:02.603389 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.603359 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2h5z\" (UniqueName: \"kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.603587 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.603461 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.603587 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.603486 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.604200 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.604179 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.605866 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.605845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.662108 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.662083 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2h5z\" (UniqueName: \"kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z\") pod \"maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:02.751219 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:02.751162 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:03.121010 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:03.120975 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7"] Nov 02 16:06:03.127240 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:06:03.127204 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc309b3a5_335c_4c49_bc52_79e7b7c23003.slice/crio-2b29f4b7eb92101e528ae56a96e23ae0d2ea3194194b0b3b4f45e549caee471e WatchSource:0}: Error finding container 2b29f4b7eb92101e528ae56a96e23ae0d2ea3194194b0b3b4f45e549caee471e: Status 404 returned error can't find the container with id 2b29f4b7eb92101e528ae56a96e23ae0d2ea3194194b0b3b4f45e549caee471e Nov 02 16:06:03.619693 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:03.619653 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" event={"ID":"c309b3a5-335c-4c49-bc52-79e7b7c23003","Type":"ContainerStarted","Data":"0c0d77a78a62cadbcf0c48e7ea6ad396b2e99aaca9279c83464d68d1b1bca216"} Nov 02 16:06:03.620062 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:03.619716 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" event={"ID":"c309b3a5-335c-4c49-bc52-79e7b7c23003","Type":"ContainerStarted","Data":"2b29f4b7eb92101e528ae56a96e23ae0d2ea3194194b0b3b4f45e549caee471e"} Nov 02 16:06:03.732377 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:03.732283 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" podStartSLOduration=1.7322696130000002 podStartE2EDuration="1.732269613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:06:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:06:03.729942337 +0000 UTC m=+4117.894494879" watchObservedRunningTime="2026-11-02 16:06:03.732269613 +0000 UTC m=+4117.896822152" Nov 02 16:06:04.626368 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:04.626334 5091 generic.go:356] "Generic (PLEG): container finished" podID="c309b3a5-335c-4c49-bc52-79e7b7c23003" containerID="0c0d77a78a62cadbcf0c48e7ea6ad396b2e99aaca9279c83464d68d1b1bca216" exitCode=0 Nov 02 16:06:04.626811 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:04.626422 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" event={"ID":"c309b3a5-335c-4c49-bc52-79e7b7c23003","Type":"ContainerDied","Data":"0c0d77a78a62cadbcf0c48e7ea6ad396b2e99aaca9279c83464d68d1b1bca216"} Nov 02 16:06:05.765779 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.765752 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:05.833391 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.833368 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca\") pod \"c309b3a5-335c-4c49-bc52-79e7b7c23003\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " Nov 02 16:06:05.833554 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.833421 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca\") pod \"c309b3a5-335c-4c49-bc52-79e7b7c23003\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " Nov 02 16:06:05.833554 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.833502 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z\" (UniqueName: \"kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z\") pod \"c309b3a5-335c-4c49-bc52-79e7b7c23003\" (UID: \"c309b3a5-335c-4c49-bc52-79e7b7c23003\") " Nov 02 16:06:05.833730 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.833709 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca" pod "c309b3a5-335c-4c49-bc52-79e7b7c23003" (UID: "c309b3a5-335c-4c49-bc52-79e7b7c23003"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:06:05.835624 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.835593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca" pod "c309b3a5-335c-4c49-bc52-79e7b7c23003" (UID: "c309b3a5-335c-4c49-bc52-79e7b7c23003"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:06:05.835624 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.835620 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z" pod "c309b3a5-335c-4c49-bc52-79e7b7c23003" (UID: "c309b3a5-335c-4c49-bc52-79e7b7c23003"). InnerVolumeSpecName "kube-api-access-x2h5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:06:05.934484 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.934397 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:05.934484 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.934438 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c309b3a5-335c-4c49-bc52-79e7b7c23003-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:05.934484 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:05.934451 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2h5z\" (UniqueName: \"kubernetes.io/projected/c309b3a5-335c-4c49-bc52-79e7b7c23003-kube-api-access-x2h5z\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:06:06.637740 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:06.637706 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" Nov 02 16:06:06.637918 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:06.637734 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7" event={"ID":"c309b3a5-335c-4c49-bc52-79e7b7c23003","Type":"ContainerDied","Data":"2b29f4b7eb92101e528ae56a96e23ae0d2ea3194194b0b3b4f45e549caee471e"} Nov 02 16:06:06.637918 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:06.637785 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b29f4b7eb92101e528ae56a96e23ae0d2ea3194194b0b3b4f45e549caee471e" Nov 02 16:06:20.556802 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:20.556764 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7"] Nov 02 16:06:20.603889 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:20.603860 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f22ac84e-shdx7"] Nov 02 16:06:21.958674 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:06:21.958640 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c309b3a5-335c-4c49-bc52-79e7b7c23003" path="/var/lib/kubelet/pods/c309b3a5-335c-4c49-bc52-79e7b7c23003/volumes" Nov 02 16:06:27.934157 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:06:27.934107 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:06:57.934798 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:06:57.934751 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:07:21.050466 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:07:21.050429 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s"] Nov 02 16:07:21.096722 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:07:21.096691 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-feb1362e-bw48s"] Nov 02 16:07:21.959049 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:07:21.959010 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e4b2d1-4db6-43fa-b02e-894571459898" path="/var/lib/kubelet/pods/f3e4b2d1-4db6-43fa-b02e-894571459898/volumes" Nov 02 16:07:27.934653 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:07:27.934572 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:07:28.950968 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:07:28.950938 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 16:07:28.976054 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:07:28.976030 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 16:07:57.934315 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:07:57.934270 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:08:17.467642 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.467599 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5"] Nov 02 16:08:17.468228 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.468205 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c309b3a5-335c-4c49-bc52-79e7b7c23003" containerName="revoke-keys" Nov 02 16:08:17.468321 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.468231 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c309b3a5-335c-4c49-bc52-79e7b7c23003" containerName="revoke-keys" Nov 02 16:08:17.468378 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.468330 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c309b3a5-335c-4c49-bc52-79e7b7c23003" containerName="revoke-keys" Nov 02 16:08:17.471390 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.471367 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.512437 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.512403 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.512576 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.512475 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6x8\" (UniqueName: \"kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.512576 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.512541 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.526502 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.526480 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5"] Nov 02 16:08:17.570771 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.570745 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:08:17.570771 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.570752 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:08:17.570941 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.570802 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:08:17.570941 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.570838 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j2bsg\"" Nov 02 16:08:17.613776 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.613748 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.613941 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.613785 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6x8\" (UniqueName: \"kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.613941 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.613811 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.614537 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.614493 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.616214 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.616195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.672361 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.672329 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6x8\" (UniqueName: \"kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8\") pod \"maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:17.784022 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:17.783938 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:18.152707 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:18.152671 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5"] Nov 02 16:08:18.158756 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:08:18.158725 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a470d01_355b_4bff_9f14_9fa82374b0c9.slice/crio-c82686a18a907816da3966ff3cbe0bdaceecb6d7057ef39a797a387ba2e5fe99 WatchSource:0}: Error finding container c82686a18a907816da3966ff3cbe0bdaceecb6d7057ef39a797a387ba2e5fe99: Status 404 returned error can't find the container with id c82686a18a907816da3966ff3cbe0bdaceecb6d7057ef39a797a387ba2e5fe99 Nov 02 16:08:18.303463 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:18.303426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" event={"ID":"9a470d01-355b-4bff-9f14-9fa82374b0c9","Type":"ContainerStarted","Data":"c82686a18a907816da3966ff3cbe0bdaceecb6d7057ef39a797a387ba2e5fe99"} Nov 02 16:08:19.310084 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:19.309998 5091 generic.go:356] "Generic (PLEG): container finished" podID="9a470d01-355b-4bff-9f14-9fa82374b0c9" containerID="3395d4e76af796d61e4b116907a7e967bd6fab91fb6162573eef577206caf2d1" exitCode=0 Nov 02 16:08:19.310449 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:19.310088 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" event={"ID":"9a470d01-355b-4bff-9f14-9fa82374b0c9","Type":"ContainerDied","Data":"3395d4e76af796d61e4b116907a7e967bd6fab91fb6162573eef577206caf2d1"} Nov 02 16:08:20.451624 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.451603 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:20.539265 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.539236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8\" (UniqueName: \"kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8\") pod \"9a470d01-355b-4bff-9f14-9fa82374b0c9\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " Nov 02 16:08:20.539404 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.539335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca\") pod \"9a470d01-355b-4bff-9f14-9fa82374b0c9\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " Nov 02 16:08:20.539404 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.539382 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca\") pod \"9a470d01-355b-4bff-9f14-9fa82374b0c9\" (UID: \"9a470d01-355b-4bff-9f14-9fa82374b0c9\") " Nov 02 16:08:20.539772 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.539747 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca" pod "9a470d01-355b-4bff-9f14-9fa82374b0c9" (UID: "9a470d01-355b-4bff-9f14-9fa82374b0c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:08:20.541277 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.541254 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8" pod "9a470d01-355b-4bff-9f14-9fa82374b0c9" (UID: "9a470d01-355b-4bff-9f14-9fa82374b0c9"). InnerVolumeSpecName "kube-api-access-2d6x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:08:20.541346 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.541311 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca" pod "9a470d01-355b-4bff-9f14-9fa82374b0c9" (UID: "9a470d01-355b-4bff-9f14-9fa82374b0c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:08:20.641076 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.641044 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d6x8\" (UniqueName: \"kubernetes.io/projected/9a470d01-355b-4bff-9f14-9fa82374b0c9-kube-api-access-2d6x8\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:20.641076 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.641073 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:20.641282 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:20.641084 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a470d01-355b-4bff-9f14-9fa82374b0c9-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:08:21.321210 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:21.321172 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" event={"ID":"9a470d01-355b-4bff-9f14-9fa82374b0c9","Type":"ContainerDied","Data":"c82686a18a907816da3966ff3cbe0bdaceecb6d7057ef39a797a387ba2e5fe99"} Nov 02 16:08:21.321368 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:21.321220 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c82686a18a907816da3966ff3cbe0bdaceecb6d7057ef39a797a387ba2e5fe99" Nov 02 16:08:21.321368 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:21.321189 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5" Nov 02 16:08:27.933459 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:08:27.933420 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:08:30.841714 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:30.841680 5091 scope.go:122] "RemoveContainer" containerID="1b1a2c2b5f90af89c568047472b4795b6096f01386cd1373a8879aa2ed0e2beb" Nov 02 16:08:30.850613 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:30.850590 5091 scope.go:122] "RemoveContainer" containerID="216d32d0ed19adac640c7b9f477e40dd89fa38842b498bae47b2dbe2931bdc53" Nov 02 16:08:39.794980 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:39.794892 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5"] Nov 02 16:08:39.841868 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:39.841834 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-715ca8-jr9v5"] Nov 02 16:08:39.959118 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:08:39.959083 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a470d01-355b-4bff-9f14-9fa82374b0c9" path="/var/lib/kubelet/pods/9a470d01-355b-4bff-9f14-9fa82374b0c9/volumes" Nov 02 16:08:57.941943 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:08:57.941907 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:09:13.847261 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.847226 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b"] Nov 02 16:09:13.847785 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.847665 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9a470d01-355b-4bff-9f14-9fa82374b0c9" containerName="revoke-keys" Nov 02 16:09:13.847785 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.847678 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a470d01-355b-4bff-9f14-9fa82374b0c9" containerName="revoke-keys" Nov 02 16:09:13.847785 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.847770 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a470d01-355b-4bff-9f14-9fa82374b0c9" containerName="revoke-keys" Nov 02 16:09:13.850893 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.850876 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:13.955007 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.954975 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:09:13.955007 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.954975 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j2bsg\"" Nov 02 16:09:13.955007 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.954986 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:09:13.955427 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.955029 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:09:13.958471 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.958445 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b"] Nov 02 16:09:13.969970 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.969949 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:13.970045 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.969994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzq2x\" (UniqueName: \"kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:13.970045 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:13.970020 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.071426 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.071387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.071628 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.071474 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzq2x\" (UniqueName: \"kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.071628 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.071553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.072168 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.072142 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.074098 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.074078 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.129950 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.129925 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzq2x\" (UniqueName: \"kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x\") pod \"maas-api-revoke-sub-f03629db-wvq5b\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.163700 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.163676 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:14.532488 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.532407 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b"] Nov 02 16:09:14.538600 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:09:14.538572 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod979c439d_77d3_4576_ac43_ae6da26e5875.slice/crio-7344ff9d09f43997726dffe657f7b93cecf5a7a50089aa12c0f557e00aa81f70 WatchSource:0}: Error finding container 7344ff9d09f43997726dffe657f7b93cecf5a7a50089aa12c0f557e00aa81f70: Status 404 returned error can't find the container with id 7344ff9d09f43997726dffe657f7b93cecf5a7a50089aa12c0f557e00aa81f70 Nov 02 16:09:14.608278 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:14.608251 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" event={"ID":"979c439d-77d3-4576-ac43-ae6da26e5875","Type":"ContainerStarted","Data":"7344ff9d09f43997726dffe657f7b93cecf5a7a50089aa12c0f557e00aa81f70"} Nov 02 16:09:15.614237 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:15.614204 5091 generic.go:356] "Generic (PLEG): container finished" podID="979c439d-77d3-4576-ac43-ae6da26e5875" containerID="11e1752b048f38217c38ed11943de2686726a5d9a7392e49e98af4cae0f44df6" exitCode=0 Nov 02 16:09:15.614652 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:15.614259 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" event={"ID":"979c439d-77d3-4576-ac43-ae6da26e5875","Type":"ContainerDied","Data":"11e1752b048f38217c38ed11943de2686726a5d9a7392e49e98af4cae0f44df6"} Nov 02 16:09:16.757895 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.757863 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:16.899750 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.899658 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x\" (UniqueName: \"kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x\") pod \"979c439d-77d3-4576-ac43-ae6da26e5875\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " Nov 02 16:09:16.899750 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.899718 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca\") pod \"979c439d-77d3-4576-ac43-ae6da26e5875\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " Nov 02 16:09:16.899996 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.899859 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca\") pod \"979c439d-77d3-4576-ac43-ae6da26e5875\" (UID: \"979c439d-77d3-4576-ac43-ae6da26e5875\") " Nov 02 16:09:16.900255 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.900223 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca" pod "979c439d-77d3-4576-ac43-ae6da26e5875" (UID: "979c439d-77d3-4576-ac43-ae6da26e5875"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:09:16.902181 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.902156 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x" pod "979c439d-77d3-4576-ac43-ae6da26e5875" (UID: "979c439d-77d3-4576-ac43-ae6da26e5875"). InnerVolumeSpecName "kube-api-access-rzq2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:09:16.902642 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:16.902622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca" pod "979c439d-77d3-4576-ac43-ae6da26e5875" (UID: "979c439d-77d3-4576-ac43-ae6da26e5875"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:09:17.001250 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:17.001214 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:17.001250 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:17.001244 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzq2x\" (UniqueName: \"kubernetes.io/projected/979c439d-77d3-4576-ac43-ae6da26e5875-kube-api-access-rzq2x\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:17.001446 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:17.001258 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/979c439d-77d3-4576-ac43-ae6da26e5875-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:17.626308 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:17.626274 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" Nov 02 16:09:17.626502 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:17.626303 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-wvq5b" event={"ID":"979c439d-77d3-4576-ac43-ae6da26e5875","Type":"ContainerDied","Data":"7344ff9d09f43997726dffe657f7b93cecf5a7a50089aa12c0f557e00aa81f70"} Nov 02 16:09:17.626502 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:17.626361 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7344ff9d09f43997726dffe657f7b93cecf5a7a50089aa12c0f557e00aa81f70" Nov 02 16:09:27.939918 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:09:27.939873 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:09:34.468060 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.468023 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz"] Nov 02 16:09:34.468447 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.468427 5091 cpu_manager.go:414] "containerMap: removing container" podUID="979c439d-77d3-4576-ac43-ae6da26e5875" containerName="revoke-keys" Nov 02 16:09:34.468447 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.468437 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="979c439d-77d3-4576-ac43-ae6da26e5875" containerName="revoke-keys" Nov 02 16:09:34.468548 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.468507 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="979c439d-77d3-4576-ac43-ae6da26e5875" containerName="revoke-keys" Nov 02 16:09:34.471788 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.471770 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.524152 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.524130 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:09:34.570604 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.570578 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:09:34.570778 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.570722 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:09:34.570867 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.570802 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j2bsg\"" Nov 02 16:09:34.572896 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.572873 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz"] Nov 02 16:09:34.602599 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.602568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.602725 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.602629 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.602725 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.602707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhtgs\" (UniqueName: \"kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.703667 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.703640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.703799 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.703716 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.703799 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.703735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhtgs\" (UniqueName: \"kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.704277 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.704258 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.706090 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.706068 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.763094 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.763024 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhtgs\" (UniqueName: \"kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs\") pod \"maas-api-revoke-sub-0165d7d2-vcscz\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:34.784867 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:34.784842 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:35.157350 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:35.157312 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz"] Nov 02 16:09:35.162919 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:09:35.162891 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99bf26fb_6cc1_44b5_b7ec_093af336c890.slice/crio-9c517da7922216bf5f0dcd454a7abf06ba8b010bd3551a6b2a15895083745edf WatchSource:0}: Error finding container 9c517da7922216bf5f0dcd454a7abf06ba8b010bd3551a6b2a15895083745edf: Status 404 returned error can't find the container with id 9c517da7922216bf5f0dcd454a7abf06ba8b010bd3551a6b2a15895083745edf Nov 02 16:09:35.721711 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:35.721631 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" event={"ID":"99bf26fb-6cc1-44b5-b7ec-093af336c890","Type":"ContainerStarted","Data":"cd173561a8f0199b2da4a37d2590cad56a289ff34776b3c97ab17a34efbe1067"} Nov 02 16:09:35.721711 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:35.721688 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" event={"ID":"99bf26fb-6cc1-44b5-b7ec-093af336c890","Type":"ContainerStarted","Data":"9c517da7922216bf5f0dcd454a7abf06ba8b010bd3551a6b2a15895083745edf"} Nov 02 16:09:35.834105 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:35.834054 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" podStartSLOduration=1.8340405579999999 podStartE2EDuration="1.834040558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:09:35.832157224 +0000 UTC m=+4329.996709764" watchObservedRunningTime="2026-11-02 16:09:35.834040558 +0000 UTC m=+4329.998593098" Nov 02 16:09:36.727532 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:36.727483 5091 generic.go:356] "Generic (PLEG): container finished" podID="99bf26fb-6cc1-44b5-b7ec-093af336c890" containerID="cd173561a8f0199b2da4a37d2590cad56a289ff34776b3c97ab17a34efbe1067" exitCode=0 Nov 02 16:09:36.727963 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:36.727533 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" event={"ID":"99bf26fb-6cc1-44b5-b7ec-093af336c890","Type":"ContainerDied","Data":"cd173561a8f0199b2da4a37d2590cad56a289ff34776b3c97ab17a34efbe1067"} Nov 02 16:09:37.868708 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.868684 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:37.937941 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.937908 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca\") pod \"99bf26fb-6cc1-44b5-b7ec-093af336c890\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " Nov 02 16:09:37.938106 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.937979 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs\" (UniqueName: \"kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs\") pod \"99bf26fb-6cc1-44b5-b7ec-093af336c890\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " Nov 02 16:09:37.938106 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.938000 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca\") pod \"99bf26fb-6cc1-44b5-b7ec-093af336c890\" (UID: \"99bf26fb-6cc1-44b5-b7ec-093af336c890\") " Nov 02 16:09:37.938377 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.938353 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca" pod "99bf26fb-6cc1-44b5-b7ec-093af336c890" (UID: "99bf26fb-6cc1-44b5-b7ec-093af336c890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:09:37.940118 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.940096 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca" pod "99bf26fb-6cc1-44b5-b7ec-093af336c890" (UID: "99bf26fb-6cc1-44b5-b7ec-093af336c890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:09:37.940118 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:37.940105 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs" pod "99bf26fb-6cc1-44b5-b7ec-093af336c890" (UID: "99bf26fb-6cc1-44b5-b7ec-093af336c890"). InnerVolumeSpecName "kube-api-access-zhtgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:09:38.038651 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:38.038570 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:38.038651 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:38.038598 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhtgs\" (UniqueName: \"kubernetes.io/projected/99bf26fb-6cc1-44b5-b7ec-093af336c890-kube-api-access-zhtgs\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:38.038651 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:38.038610 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99bf26fb-6cc1-44b5-b7ec-093af336c890-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:38.738090 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:38.738052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" event={"ID":"99bf26fb-6cc1-44b5-b7ec-093af336c890","Type":"ContainerDied","Data":"9c517da7922216bf5f0dcd454a7abf06ba8b010bd3551a6b2a15895083745edf"} Nov 02 16:09:38.738090 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:38.738077 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-vcscz" Nov 02 16:09:38.738370 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:38.738104 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c517da7922216bf5f0dcd454a7abf06ba8b010bd3551a6b2a15895083745edf" Nov 02 16:09:46.148564 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.148529 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc"] Nov 02 16:09:46.149035 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.148910 5091 cpu_manager.go:414] "containerMap: removing container" podUID="99bf26fb-6cc1-44b5-b7ec-093af336c890" containerName="revoke-keys" Nov 02 16:09:46.149035 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.148921 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99bf26fb-6cc1-44b5-b7ec-093af336c890" containerName="revoke-keys" Nov 02 16:09:46.149035 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.149002 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="99bf26fb-6cc1-44b5-b7ec-093af336c890" containerName="revoke-keys" Nov 02 16:09:46.152349 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.152327 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.213918 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.213886 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtc4w\" (UniqueName: \"kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.214103 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.213931 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.214103 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.213963 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.220307 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.220282 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc"] Nov 02 16:09:46.264467 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.264432 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 16:09:46.264467 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.264432 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 16:09:46.264681 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.264469 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 16:09:46.264681 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.264492 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-j2bsg\"" Nov 02 16:09:46.315357 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.315324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtc4w\" (UniqueName: \"kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.315357 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.315360 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.315612 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.315382 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.316032 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.316011 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.317845 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.317818 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.374434 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.374399 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtc4w\" (UniqueName: \"kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w\") pod \"maas-api-revoke-sub-6b19338f-d6vdc\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.464542 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.464447 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:46.835834 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:46.835735 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc"] Nov 02 16:09:46.841698 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:09:46.841664 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec06b64c_4ecc_4640_aa94_88b809d01518.slice/crio-b5b141a0525bc821e220a73170d34a619c8e3f89892027b26cb1aa52a75143a3 WatchSource:0}: Error finding container b5b141a0525bc821e220a73170d34a619c8e3f89892027b26cb1aa52a75143a3: Status 404 returned error can't find the container with id b5b141a0525bc821e220a73170d34a619c8e3f89892027b26cb1aa52a75143a3 Nov 02 16:09:47.785048 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:47.785017 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec06b64c-4ecc-4640-aa94-88b809d01518" containerID="9ec915bdfcbbe7813e775c020acc0e555f0075c837bf5e3c5da26624f7288b22" exitCode=0 Nov 02 16:09:47.785427 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:47.785055 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" event={"ID":"ec06b64c-4ecc-4640-aa94-88b809d01518","Type":"ContainerDied","Data":"9ec915bdfcbbe7813e775c020acc0e555f0075c837bf5e3c5da26624f7288b22"} Nov 02 16:09:47.785427 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:47.785119 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" event={"ID":"ec06b64c-4ecc-4640-aa94-88b809d01518","Type":"ContainerStarted","Data":"b5b141a0525bc821e220a73170d34a619c8e3f89892027b26cb1aa52a75143a3"} Nov 02 16:09:48.929372 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:48.929348 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:49.044983 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.044889 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w\" (UniqueName: \"kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w\") pod \"ec06b64c-4ecc-4640-aa94-88b809d01518\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " Nov 02 16:09:49.045160 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.045000 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca\") pod \"ec06b64c-4ecc-4640-aa94-88b809d01518\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " Nov 02 16:09:49.045160 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.045080 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca\") pod \"ec06b64c-4ecc-4640-aa94-88b809d01518\" (UID: \"ec06b64c-4ecc-4640-aa94-88b809d01518\") " Nov 02 16:09:49.045618 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.045589 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca" pod "ec06b64c-4ecc-4640-aa94-88b809d01518" (UID: "ec06b64c-4ecc-4640-aa94-88b809d01518"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 16:09:49.047724 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.047693 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w" pod "ec06b64c-4ecc-4640-aa94-88b809d01518" (UID: "ec06b64c-4ecc-4640-aa94-88b809d01518"). InnerVolumeSpecName "kube-api-access-rtc4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 16:09:49.047852 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.047764 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca" pod "ec06b64c-4ecc-4640-aa94-88b809d01518" (UID: "ec06b64c-4ecc-4640-aa94-88b809d01518"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 16:09:49.146011 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.145984 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-xks-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:49.146011 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.146010 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec06b64c-4ecc-4640-aa94-88b809d01518-maas-api-openshift-ca\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:49.146195 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.146021 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtc4w\" (UniqueName: \"kubernetes.io/projected/ec06b64c-4ecc-4640-aa94-88b809d01518-kube-api-access-rtc4w\") on node \"ip-10-0-130-229.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 16:09:49.798029 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.797994 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" Nov 02 16:09:49.798217 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.797994 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-d6vdc" event={"ID":"ec06b64c-4ecc-4640-aa94-88b809d01518","Type":"ContainerDied","Data":"b5b141a0525bc821e220a73170d34a619c8e3f89892027b26cb1aa52a75143a3"} Nov 02 16:09:49.798217 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:09:49.798122 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b141a0525bc821e220a73170d34a619c8e3f89892027b26cb1aa52a75143a3" Nov 02 16:09:57.933670 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:09:57.933633 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:10:10.293679 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:10.293598 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-748vc_0243dab1-1945-4a78-9ef4-d04c99c2cb26/manager/0.log" Nov 02 16:10:11.759149 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:11.759109 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xqw9r_da5b943e-8fb9-48f3-8757-8a3306654620/proxy/0.log" Nov 02 16:10:13.146825 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:13.146797 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vcscz_99bf26fb-6cc1-44b5-b7ec-093af336c890/revoke-keys/0.log" Nov 02 16:10:13.359038 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:13.359007 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-d6vdc_ec06b64c-4ecc-4640-aa94-88b809d01518/revoke-keys/0.log" Nov 02 16:10:13.571657 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:13.571580 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wvq5b_979c439d-77d3-4576-ac43-ae6da26e5875/revoke-keys/0.log" Nov 02 16:10:15.787684 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:15.787654 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zt62h_07c22d6f-df0c-4478-9683-48ba21cf1493/manager/0.log" Nov 02 16:10:16.015876 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:16.015847 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-r8wss_86aa6b27-08ca-42e7-82ad-4347bef0c37d/manager/0.log" Nov 02 16:10:16.497947 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:16.497914 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bhprn_d6d2801c-b13e-4aca-bf65-0ca8807954dc/manager/0.log" Nov 02 16:10:17.811675 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:17.811638 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc_a43c1886-7c1c-48a9-877e-91d7a4bd6a8d/istio-proxy/0.log" Nov 02 16:10:19.930019 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:19.929983 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pqg4x_30c3614d-aa3f-45ed-af95-c4d527cd3f57/istio-proxy/0.log" Nov 02 16:10:20.464395 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:20.464355 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r2p85_ee194769-7328-456a-b24d-c368212d5672/istio-proxy/0.log" Nov 02 16:10:21.220317 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:21.220279 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xv6t5_02ebee9d-d52a-4ff9-a0b3-3f071539045d/istio-proxy/0.log" Nov 02 16:10:22.124580 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:22.124508 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54fb4b65f9-w4zp9_108f7147-992f-44ce-8cc4-dd7f8cd2521d/router/0.log" Nov 02 16:10:27.933377 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:10:27.933329 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:10:35.623565 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:35.623511 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-748vc_0243dab1-1945-4a78-9ef4-d04c99c2cb26/manager/0.log" Nov 02 16:10:36.809249 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:36.809199 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-xqw9r_da5b943e-8fb9-48f3-8757-8a3306654620/proxy/0.log" Nov 02 16:10:37.919612 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:37.919578 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-vcscz_99bf26fb-6cc1-44b5-b7ec-093af336c890/revoke-keys/0.log" Nov 02 16:10:38.136147 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:38.136112 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-d6vdc_ec06b64c-4ecc-4640-aa94-88b809d01518/revoke-keys/0.log" Nov 02 16:10:38.358766 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:38.358681 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-wvq5b_979c439d-77d3-4576-ac43-ae6da26e5875/revoke-keys/0.log" Nov 02 16:10:40.202989 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:40.202953 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zt62h_07c22d6f-df0c-4478-9683-48ba21cf1493/manager/0.log" Nov 02 16:10:40.438249 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:40.438211 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-r8wss_86aa6b27-08ca-42e7-82ad-4347bef0c37d/manager/0.log" Nov 02 16:10:40.864226 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:40.864185 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bhprn_d6d2801c-b13e-4aca-bf65-0ca8807954dc/manager/0.log" Nov 02 16:10:41.951384 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:41.951354 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc_a43c1886-7c1c-48a9-877e-91d7a4bd6a8d/istio-proxy/0.log" Nov 02 16:10:43.939531 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:43.939493 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pqg4x_30c3614d-aa3f-45ed-af95-c4d527cd3f57/istio-proxy/0.log" Nov 02 16:10:44.457992 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:44.457951 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r2p85_ee194769-7328-456a-b24d-c368212d5672/istio-proxy/0.log" Nov 02 16:10:45.164727 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:45.164692 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-xv6t5_02ebee9d-d52a-4ff9-a0b3-3f071539045d/istio-proxy/0.log" Nov 02 16:10:46.044607 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:10:46.044576 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54fb4b65f9-w4zp9_108f7147-992f-44ce-8cc4-dd7f8cd2521d/router/0.log" Nov 02 16:10:57.936652 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:10:57.936610 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:11:27.934802 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:11:27.934757 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:11:54.832354 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.832241 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2cdf/must-gather-pn58t"] Nov 02 16:11:54.832945 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.832708 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec06b64c-4ecc-4640-aa94-88b809d01518" containerName="revoke-keys" Nov 02 16:11:54.832945 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.832722 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec06b64c-4ecc-4640-aa94-88b809d01518" containerName="revoke-keys" Nov 02 16:11:54.832945 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.832780 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec06b64c-4ecc-4640-aa94-88b809d01518" containerName="revoke-keys" Nov 02 16:11:54.836166 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.836140 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:54.892127 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.892072 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cdf\"/\"openshift-service-ca.crt\"" Nov 02 16:11:54.904976 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.904935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51948afd-8d8e-4336-af5a-3b5239eac034-must-gather-output\") pod \"must-gather-pn58t\" (UID: \"51948afd-8d8e-4336-af5a-3b5239eac034\") " pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:54.905219 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.905145 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79xs7\" (UniqueName: \"kubernetes.io/projected/51948afd-8d8e-4336-af5a-3b5239eac034-kube-api-access-79xs7\") pod \"must-gather-pn58t\" (UID: \"51948afd-8d8e-4336-af5a-3b5239eac034\") " pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:54.944328 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.944290 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f2cdf\"/\"kube-root-ca.crt\"" Nov 02 16:11:54.944578 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.944354 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f2cdf\"/\"default-dockercfg-bcp6z\"" Nov 02 16:11:54.946703 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:54.946669 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/must-gather-pn58t"] Nov 02 16:11:55.006853 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.006798 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79xs7\" (UniqueName: \"kubernetes.io/projected/51948afd-8d8e-4336-af5a-3b5239eac034-kube-api-access-79xs7\") pod \"must-gather-pn58t\" (UID: \"51948afd-8d8e-4336-af5a-3b5239eac034\") " pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:55.006853 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.006865 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51948afd-8d8e-4336-af5a-3b5239eac034-must-gather-output\") pod \"must-gather-pn58t\" (UID: \"51948afd-8d8e-4336-af5a-3b5239eac034\") " pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:55.007275 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.007250 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/51948afd-8d8e-4336-af5a-3b5239eac034-must-gather-output\") pod \"must-gather-pn58t\" (UID: \"51948afd-8d8e-4336-af5a-3b5239eac034\") " pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:55.070971 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.070928 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79xs7\" (UniqueName: \"kubernetes.io/projected/51948afd-8d8e-4336-af5a-3b5239eac034-kube-api-access-79xs7\") pod \"must-gather-pn58t\" (UID: \"51948afd-8d8e-4336-af5a-3b5239eac034\") " pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:55.167416 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.167375 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/must-gather-pn58t" Nov 02 16:11:55.552068 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.552033 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/must-gather-pn58t"] Nov 02 16:11:55.552773 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:11:55.552740 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51948afd_8d8e_4336_af5a_3b5239eac034.slice/crio-2027bdbb3dc7c2936f8d3d2f65d3cb97f8f775757387968dbe9b1cdd44ecef79 WatchSource:0}: Error finding container 2027bdbb3dc7c2936f8d3d2f65d3cb97f8f775757387968dbe9b1cdd44ecef79: Status 404 returned error can't find the container with id 2027bdbb3dc7c2936f8d3d2f65d3cb97f8f775757387968dbe9b1cdd44ecef79 Nov 02 16:11:55.554906 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:55.554882 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 16:11:56.501089 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:56.501048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/must-gather-pn58t" event={"ID":"51948afd-8d8e-4336-af5a-3b5239eac034","Type":"ContainerStarted","Data":"2027bdbb3dc7c2936f8d3d2f65d3cb97f8f775757387968dbe9b1cdd44ecef79"} Nov 02 16:11:57.510617 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:57.510575 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/must-gather-pn58t" event={"ID":"51948afd-8d8e-4336-af5a-3b5239eac034","Type":"ContainerStarted","Data":"d4774e62d05190e4a2f0b7dd0a2a0d56b4ff080519741315c0d139f427585268"} Nov 02 16:11:57.511175 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:57.510635 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/must-gather-pn58t" event={"ID":"51948afd-8d8e-4336-af5a-3b5239eac034","Type":"ContainerStarted","Data":"a9d6e3b2a992afa953be91e9dad67a81965e9d7aad5d9440328f0e384b59e41e"} Nov 02 16:11:57.633202 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:11:57.633136 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-f2cdf/must-gather-pn58t" podStartSLOduration=2.421842619 podStartE2EDuration="3.633122494s" totalImagesPullingTime="1.211279875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 16:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:11:57.63254844 +0000 UTC m=+4471.797100981" watchObservedRunningTime="2026-11-02 16:11:57.633122494 +0000 UTC m=+4471.797675034" Nov 02 16:11:57.937647 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:11:57.937607 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:12:03.946486 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:03.946430 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-88g4t_a3e70c22-8486-4834-8332-a8cf3e34d738/global-pull-secret-syncer/0.log" Nov 02 16:12:05.116541 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:05.116223 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7hpmc_a88a6270-a93e-487d-a0b2-53a5b020c388/connection-checker/0.log" Nov 02 16:12:05.522542 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:05.520892 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-7s4b6_f264a363-68bf-4fc1-87c6-824a8796d7ef/connection-checker/0.log" Nov 02 16:12:05.922538 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:05.921979 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-wfz5w_e66a1a72-e152-44d7-aa2b-846e8a872078/connection-checker/0.log" Nov 02 16:12:06.925601 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:06.925555 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-48fls_8b68b41e-2bcf-4d2c-a881-dcd809b89fc5/konnectivity-agent/0.log" Nov 02 16:12:07.728567 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:07.723721 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-229.us-west-2.compute.internal_548a2bc445bef7413f406b710579c42a/haproxy/0.log" Nov 02 16:12:13.707921 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:13.707873 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-zt62h_07c22d6f-df0c-4478-9683-48ba21cf1493/manager/0.log" Nov 02 16:12:14.104585 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:14.104458 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-r8wss_86aa6b27-08ca-42e7-82ad-4347bef0c37d/manager/0.log" Nov 02 16:12:14.954748 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:14.954710 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-bhprn_d6d2801c-b13e-4aca-bf65-0ca8807954dc/manager/0.log" Nov 02 16:12:14.997376 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:14.997326 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds"] Nov 02 16:12:15.006216 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.006183 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.057096 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.057044 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds"] Nov 02 16:12:15.081806 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.081752 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbstw\" (UniqueName: \"kubernetes.io/projected/0d2c7011-0834-4b49-8ada-b2114a6f7530-kube-api-access-gbstw\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.082022 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.081889 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-sys\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.082022 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.081968 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-proc\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.082022 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.082002 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-podres\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.082239 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.082214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-lib-modules\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184033 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.183985 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-lib-modules\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184058 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbstw\" (UniqueName: \"kubernetes.io/projected/0d2c7011-0834-4b49-8ada-b2114a6f7530-kube-api-access-gbstw\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184098 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-sys\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184123 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-proc\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184148 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-podres\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184196 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-lib-modules\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184223 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-proc\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184276 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184212 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-sys\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.184604 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.184318 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0d2c7011-0834-4b49-8ada-b2114a6f7530-podres\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.248637 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.248502 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbstw\" (UniqueName: \"kubernetes.io/projected/0d2c7011-0834-4b49-8ada-b2114a6f7530-kube-api-access-gbstw\") pod \"perf-node-gather-daemonset-m5nds\" (UID: \"0d2c7011-0834-4b49-8ada-b2114a6f7530\") " pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.321158 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.321103 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:15.718657 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:15.718620 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds"] Nov 02 16:12:15.722424 ip-10-0-130-229 kubenswrapper[5091]: W1102 16:12:15.722383 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0d2c7011_0834_4b49_8ada_b2114a6f7530.slice/crio-7d1dbe80976909190cea3167be5fd430e405b0d8a9ab5916143bc55f1f7b3465 WatchSource:0}: Error finding container 7d1dbe80976909190cea3167be5fd430e405b0d8a9ab5916143bc55f1f7b3465: Status 404 returned error can't find the container with id 7d1dbe80976909190cea3167be5fd430e405b0d8a9ab5916143bc55f1f7b3465 Nov 02 16:12:16.710034 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:16.709993 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" event={"ID":"0d2c7011-0834-4b49-8ada-b2114a6f7530","Type":"ContainerStarted","Data":"4d04d4ed71292b978b5cb056fb6a5dee056f18b1d07f640dfd12b649d480383e"} Nov 02 16:12:16.710633 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:16.710100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" event={"ID":"0d2c7011-0834-4b49-8ada-b2114a6f7530","Type":"ContainerStarted","Data":"7d1dbe80976909190cea3167be5fd430e405b0d8a9ab5916143bc55f1f7b3465"} Nov 02 16:12:16.710633 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:16.710135 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:16.832792 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:16.832722 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" podStartSLOduration=2.83270747 podStartE2EDuration="2.83270747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 16:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 16:12:16.831673251 +0000 UTC m=+4490.996225791" watchObservedRunningTime="2026-11-02 16:12:16.83270747 +0000 UTC m=+4490.997260010" Nov 02 16:12:21.160375 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:21.160333 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-w8df7_8e6e6852-e747-4869-aea2-5e0bbc44988f/cluster-monitoring-operator/0.log" Nov 02 16:12:22.733869 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:22.733840 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f2cdf/perf-node-gather-daemonset-m5nds" Nov 02 16:12:25.874672 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:25.874628 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zhvgg_49c54e7f-83d6-4b42-a6e9-f4494a15206f/node-exporter/0.log" Nov 02 16:12:26.274008 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:26.273923 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zhvgg_49c54e7f-83d6-4b42-a6e9-f4494a15206f/kube-rbac-proxy/0.log" Nov 02 16:12:26.664941 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:26.664910 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zhvgg_49c54e7f-83d6-4b42-a6e9-f4494a15206f/init-textfile/0.log" Nov 02 16:12:27.937447 ip-10-0-130-229 kubenswrapper[5091]: E1102 16:12:27.937402 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r2l82" containerName="extractor" Nov 02 16:12:28.999292 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:28.999256 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 16:12:29.026354 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:29.026324 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-8vstz_dd9fb448-f983-484e-85a4-afbd3efddd1d/console-operator/2.log" Nov 02 16:12:30.901817 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:30.901765 5091 scope.go:122] "RemoveContainer" containerID="0c0d77a78a62cadbcf0c48e7ea6ad396b2e99aaca9279c83464d68d1b1bca216" Nov 02 16:12:31.883610 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:31.883577 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2zfvw_6fc6839b-0cee-4266-ad3b-c9355dfd01e8/prometheus-operator-admission-webhook/0.log" Nov 02 16:12:37.653300 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:37.653254 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7cfc97f8f78d6wc_a43c1886-7c1c-48a9-877e-91d7a4bd6a8d/istio-proxy/0.log" Nov 02 16:12:38.348484 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:38.348448 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-nsrp7_2b735564-c5ef-4cc9-838c-6bd04cb9dc50/networking-console-plugin/0.log" Nov 02 16:12:39.510812 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:39.510776 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-pqg4x_30c3614d-aa3f-45ed-af95-c4d527cd3f57/istio-proxy/0.log" Nov 02 16:12:39.936753 ip-10-0-130-229 kubenswrapper[5091]: I1102 16:12:39.936720 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r2p85_ee194769-7328-456a-b24d-c368212d5672/istio-proxy/0.log"